Provisioning Disaster Recovery with ZFS, iSCSI and VMware

OpenSolaris, ZFS, iSCSI and VMware are a great combination for provisioning Disaster Recovery (DR) systems at exceptionally low cost. There are some fundamentally well suited features of ZFS and VMFS volumes that provide a relatively simply and very efficient recovery process for VMware hosted non-zero RPO crash consistent recovery based environments. In this weblog I will demonstrate this capability and provide some step by step howto’s for replicating a ZFS, iSCSI and VMFS VMware based environment securely over a WAN or whatever you may have to a single ESXi remote server hosting a child OpenSolaris VM which provisions ZFS and iSCSI VMFS LUN’s to the parent ESXi host. The concept is to wrap the DR services into a single low cost self contained DR box that can be expanded out in the event of an actual DR incident while allowing for regular testing and validation processing without the high costs normally associated with standby DR systems. As one would expect this method becomes a very appealing solution for small to medium businesses who would normally abstain from DR provisioning activity due to the inherently high cost and complexity of DR.

The following diagram illustrates this DR system architecture.

 

 Disaster Recovery Architecture by Mike La Spina

When we have VMFS volumes backed by iSCSI based ZFS targets we gain the powerful replication capability of ZFS send and receive commands. This ZFS feature procures the ability to send an entire VMFS volume by way of a raw iSCSI target ZFS backing store. And once sent initially we can base all subsequent sends as a delta of change from a previous send snapshot which are respectively referred as snapshot deltas. Thus if we initially snapshot an iSCSI backing store and send the stream to a remote ZFS file system we can then send all the changed object data from that previous snapshot point to the current snapshot point and whatever else may be in between those snapshots. The result is a constant update of VMFS changes from the source ZFS file system to the remote ZFS file system of which can be completely different hardware. This ZFS hardware autonomy gift allows us to provision a much lower cost system on the DR remote side to host the VMFS volumes. For example the target system which is presented in this weblog is an IBM x3500 and the source is a SUN X4500 detailed in a previous blog.

There are some important considerations that should be kept in mind when we use snapshots to create any DR process. One of the most important areas to consider is the data change rates on the VMFS volumes that are to be included in the DR send/receive process. When we have VMware servers or VM’s that have low memory allocations (a.k.a. over committed memory) or application behaviors that swap to disk frequently we will observe high volumes of what I call disk noise or disk data change that has no permanent value. High amounts of disk noise will consume more storage and bandwidth on both systems when snapshots are present. In cases where the disk noise reaches a rate of 1GB/Day or more per volume it would be prudent to isolate the noise sources on a VMFS volume that will not be part of the replication strategy. You could for example create a VMFS LUN for swap and temp files on the local ESX host which can be ignored in the replication scope. Another important area is the growth rate of the overall storage may require routine pruning of older snapshots to reduce the total consumption of disk. For example if we have high change rates from database sources which can not be isolated we can at monthly intervals destroy all but one of the last months snapshots to conserve the available storage on both systems. This method still provisions a good DR process and as well provides a level of continuous data protection (CDP) and is simmilar to a grandfather/father/son preservation scheme.

Since we are handling valuable information we must use secure methods to access and handle the data transfers. This can be provisioned by using ssh and dedicated service accounts that will perform this one specific function. ZFS send and receive functions use an inherently secure approach by employing ssh as a transport tunnel when transmitting storage data to the target ZFS file system. This is just what we need to provision a secure exchange to a DR environment. Conversly we could use IPSec but this would be significantly more complex to achieve and complexity is not a good this when short implementation time is a priority.With this explanation wrapped up in our minds lets begin some of the detailed tasks that are required to procure this real world DR solution.

ESXi Server  

The DR VMware server is based on the free ESXi product and with this we can encapsulate the entire DR functionallity in one server hardware platform. Within the ESXi engine we need to install and configure an OpenSolaris 5.11 snv_98 or higher VM using VMFS as the storage provider. This ESXi server configuration consists of a single SATA boot LUN and this LUN also stores the OpenSolaris iSCSI VM. In addition to the boot LUN we will create the ZFS iSCSI stores on serveral additional SATA disks that are presented to the OpenSolaris VM as separate VMFS datastores which we will use to create large vmdk’s. The virtual vmdk disks will be assigned as vdev’s for our receiving ZFS zpool. Talk about rampent layering. At this point we have a OpenSolaris VM defined on a hardware platform on which OpenSolaris would normally never work with natively in this example. You have goto love what you can do with VMware virtualization. By the way when SUN’s xVM product is more mature it could provision the same fuctionallity with native ZFS provisioning and that alone really is worth a talk, but lets continue our focus on this platform for now.

There are many configuration options available on the network provisioning side of our ESXi host. In this case VLAN’s are definetly a solid choice for this application and is my prefered approach to controlling iSCSI data flow. We initially would only need to provide iSCSI access for the local OpenSolaris VM as this will provision a virtual SAN to the parent ESXi host. The parent ESXi host needs to be able mount the iSCSI target LUN’s that were available in the production environmant and validate that the DR process works. In the event of DR activation we would need to add external ESXi hosts and VLAN’s will provide both locally isolated iSCSI networks with easy expansion if these services are required externally all with out need to purchase external switch hardware for the system until it is required. Thus within the ESXi host we need to define a VLAN for the iSCSI SAN and an isolated VLAN for production VM validations and finally we need to define the replication and management network which can optionally use a VLAN or be untagged depending on your environment.

This virtualized DR environment grants advanced capabilties to perform rich system tests at commodity prices. Very attracive indeed. For example you can now clone the replicated VMFS LUN’s on the DR engine and with a liitle Solaris SMF iSCSI target service magic provision the clone as a duplicated ESX environment which does not impact the ongoing replication. As well we have network isolation and virtualization that allows the environment to exist in a closed fully functional remotely accessible world. This world can also be extended out as a production mirror test environment with dynamic revert back in time and repeat functionallity.

There are many possible ESXi network and disk configurations that would meet the DR server’s requirements. At the minimum we should provision the following elements.

  •  Provision a bootable single separate SATA disk with a minimum of 16G available for the VMFS LUN that will store the OpenSolaris iSCSI VM.
  •  Provision a minimum of three (optimally six) additional SATA disks or more if required as VMFS LUN’s to host the ZFS zpool vdev’s with vmdk’s.
  •  Provision a minimum of two 1Gb Ethernet adaptors, teamed would be preferable if more are available.
  •  Define vSwitch0 with a VLAN tagged VM Network portgroup to connect the replication side of the OpenSolaris iSCSI VM and a Service Console portgroup to manage the ESXi host.
  •  Define vSwitch1 with a VLAN tagged iSCSI VM kernel portgroup to service the iSCSI data plane and also define a VM Network portgroup on the same VLAN to connect with the target interface of the OpenSolaris iSCSI VM.
  •  Define the required isolated VLAN tagged identically named portgroups  as production on vSwitch0 and use a separated VLAN numbering set for them for isolation.
  •  Define the OpenSolaris VM with one adapter to connected to the production network portgroup and one adapter to attached to the iSCSI data plane portgroup to serve the iSCSI target IP.

Here is an example of what the VM disk assignments should look like.

VMFS LUN example by Mike La Spina

Once the ESXi server is successfully built and the Opensolaris iSCSI VM is installed and functional we can create the required elements for enabling ZFS replication.

Create Service Accounts

On the systems that will act as replication partners create zfsadm ID’s as service accounts using the provided commands.

# useradd -s /usr/bin/bash -d /export/home/zfsadm -P ‘ZFS File System Management’ zfsadm
# mkdir /export/home/zfsadm
# mkdir /export/home/zfsadm/backup
# cp /etc/skel/* /export/home/zfsadm
# echo PATH=/usr/bin:/usr/sbin:/usr/ucb:/etc:. > /export/home/zfsadm/.profile
# echo export PATH >> /export/home/zfsadm/.profile
# chown –R zfsadm /export/home/zfsadm
# passwd zfsadm

Note the parameter -P ‘ZFS File System Management’, This will grant the account an RBAC profile association to administratively manage our ZFS file system unlike root which is much too powerful and is all to often used by many of us. 

The next step is to generate some crypto keys for ssh connectivity we start this with a login as the newly created zfsadm user and run a secure shell locally to ensure you have a .ssh directory and key files created in the home drive for the zfsadm user. Note this directory is normally hidden. 

# ssh localhost
The authenticity of host ‘localhost (127.0.0.1)’ can’t be established.
RSA key fingerprint is 0c:aa:64:72:84:b5:04:1c:a2:d0:42:8e:9f:4e:09:9d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘localhost’ (RSA) to the list of known hosts.
Password:
# exit

Now that we have the .ssh directory we can create a crypto key pair and configure a relatively secure login without the need to enter a password for the remote host using this account.

Do not enter a passphrase, it needs to be blank.

# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/export/home/zfsadm/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/zfsadm/.ssh/id_dsa.
Your public key has been saved in /export/home/zfsadm/.ssh/id_dsa.pub.
The key fingerprint is:
bf:58:7b:97:8d:b5:d2:31:26:14:4c:9f:ce:72:a7:20 zfsadm@ss1

The id_dsa file should not be exposed outside of this directory as it contains the private key of the pair, only the public key file id_dsa.pub needs to be exported. Now that our key pair is generated we need to append the public portion of the key pair to a file named authorized_keys2. 

# cat $HOME/.ssh/id_dsa.pub >> $HOME/.ssh/authorized_keys2

Repeat all the Create Service Accounts steps and crypto key steps on the remote server as well.

We will use the Secure Copy command to place the public key file on each opposing hosts zfsadm users home directory so that when the ssh tunnel is started the remote host can decrypt the encrypted connection request completing the tunnel which is generated with the private part of the pair. This is why we must protect the private part of the pair from exposure. Granted we have also defined an additional layer of security here by defining a dedicated user for the ZFS send activity it is very important that the private key is secured properly and it is not necessary to back it up as you can regenerate them if required.

From the local server here named ss1 (The remote server is ss2)

# scp $HOME/.ssh/id_dsa.pub ss2:$HOME/.ssh/ss1.pub
Password:
id_dsa.pub      100% |**********************************************|   603       00:00
# scp ss2:$HOME/.ssh/id_dsa.pub $HOME/.ssh/ss2.pub
Password:
id_dsa.pub      100% |**********************************************|   603       00:00
# cat $HOME/.ssh/ss2.pub >> $HOME/.ssh/authorized_keys2

And on the remote server ss2

# ssh ss2
password:
# cat $HOME/.ssh/ss1.pub >> $HOME/.ssh/authorized_keys2
# exit

This completes the trusted key secure login configuration and you should be able to secure shell from either system to the other without a password prompt using the zfsadm account. To further limit security exposure we could employe ipaddress restrictions and as well enable a firewall but this is beyond the scope of this blog.

 Target Pool and ZFS rights

As a prerequisite you need to create the receiving zpool on the target to allow the zfs sends to occur. The receiving zpool name should be the same as the source to allow ease in the re-serving of iSCSI targets. Earlier we granted the “ZFS File System Management” profile to this zfsadm user. This RBAC profile allows us to run a pfexec command which pre checks what profiles the user is assigned and then executes appropriately based on this assignment. The bonus here is you do not have to create granular rights assignments to the ZFS file system.

On the target server create your receiveing zpool.

# zpool create rp1 <your vdev’s>

Create a Cron Job

Using a cron job we will invoke our ZFS snapshots and send tasks to the target host with the execution of a bash script named zfs-daily.sh. We need to use the crontab command to create a job that will execute it as the zfsadm user, no other user except root can access this job and that a good thing considering it has the ability to shell to another host!

As root add the zfs user name to the /etc/cron.d/cron.allow file.

# echo zfsadm >> /etc/cron.d/cron.allow
# crontab –e zfsadm
59 23 * * * ./zfs-daily.sh zfs-daily.rpl

Hint: crontab uses vi – http://www.kcomputing.com/kcvi.pdf  “vi cheat sheet”

The key sequence would be hit “i” and key in the line then hit “esc :wq” and to abort “esc :q!”

Be aware of the timezone the cron service runs under, you should check it and adjust it if required. Here is a example of whats required to set it. 

# pargs -e `pgrep -f /usr/sbin/cron`

8550:   /usr/sbin/cron
envp[0]: LOGNAME=root
envp[1]: _=/usr/sbin/cron
envp[2]: LANG=en_US.UTF-8
envp[3]: PATH=/usr/sbin:/usr/bin
envp[4]: PWD=/root
envp[5]: SMF_FMRI=svc:/system/cron:default
envp[6]: SMF_METHOD=start
envp[7]: SMF_RESTARTER=svc:/system/svc/restarter:default
envp[8]: SMF_ZONENAME=global
envp[9]: TZ=PST8PDT

Let’s change it to CST6CDT

# svccfg -s system/cron:default setenv TZ CST6DST

Also the default environment path for cron may cause some script “command not found” issues, check for a path and adjust it if required.

# cat /etc/default/cron
#
# Copyright 1991 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
#pragma ident   “%Z%%M% %I%     %E% SMI”
CRONLOG=YES

This one has no default path, add the path using echo.

# echo PATH=/usr/bin:/usr/sbin:/usr/ucb:/etc:. > /etc/default/cron
# svcadm refresh cron
# svcadm restart cron
 

Create Snapshot Replication Script

Here is the link for the zfs-daily.sh replication script you will need to grant exec rights to this file e.g.

# chmod 755 zfs-daily.sh 

The replcation script needs to live in the zfsadm home directory /export/home/zfsadm at this point I only have the one script built but other ones are in the works like a grandfather/father/son snapshot rollup script. The first run of the script can take a considerable amount of time depending on the available bandwidth and size of the VMFS luns. This cron job runs at midnight and took 6 hours over 100MB’s of bandwidth the first time and less that 5 min thereafter. A secondary script that runs hourly and is rolled up at days end would be beneficial. I will get it around to that one and the grandfather/father/son script later.

At this point we have an automated DR process that provides a form of CDP. But we do not have a way to access it so we need to perform some additional steps. In order for VMware to use the relocated VMFS iSCSI targets we need to reinstate some critical configuration info that was stored on the source Service Management Facility (SMF) repository. Within the iscsitgtd service properties we have the Network Address Authority (NAA) value which is named GUID in the properties list. This value is very important, when a VMFS is initialized the NAA is written to the VMFS volume header and this will need to be redefined on the DR target so that VMware will recognize the data store as available. If the NAA on the header and target do not match, the volume will not be visible to the DR VMware ESXi host. To protect this configuration info we need to export it from the source host and send it to the target host.

Export SMF iSCSI configuration

The iscstgtd service configuration elements can be easily exported using the following  command.

# svccfg export iscsitgt > /export/home/zfsadm/backup/ss1-iscsitgt.xml
 

Once exported to the backup directory we can Secure Copy this directory to the target system and this directory may also contain other useful info like installation instructions and so forth.

# scp ss1:/export/home/zfsadm/backup/* ss2:/export/home/zfsadm/backup/

This scp directory copy can be added to the zfs-daily.sh crontab script after it is performed once manually as it requires an interactive key signature trust authorization alternately it can be done manually after a configuration change occurs. I prefer the automated method so it is included in the script.

SMF iscsitgt import and iSCSI configuration details

To import the production service we would issue the following commands.

# svcadm disable iscsitgt
# svccfg delete iscsitgt
# svccfg import /export/home/zfsadm/backup/ss1-iscsitgt.xml

Importing the iscsitgt service configuration is a simple task but it does have some elements that will be problematic if they are left unchecked. For example iSCSI Target Portal Group Tag values are included with the exported/inport function and thus you may need to change the portal groups values to correct discovery failure when the ip addresses are different on the target system. Another potential issue is leaving the existing SMF config in place and then importing the new one on top of it. This is not a best practice as you may create an invalid SMF for the iscsitgt service with elements that are orphaned out etc. The SMF properties will have the backing store path from the source server and if the target server does not have the same zpool name this will need to be fixed. And lastly make sure you have the same iscsitgtd version on each end since it will have potential changes between the versions.

You will also need to add the ESXi software initiator to the iSCSI target(s) on the receiving server and grant access with an acl entry and chap info if used.

# iscsitadm create initiator –iqn iqn.1998-01.com.vmware:vh0.0 vh0.0
# iscsitadm modify target –acl vh0.0 ss1-zstore0

To handle a TPGT configuration change its simply a matter of re-adding them with the iscsitadm utility as demonstrated here or possibly deleting the one that are not correct.

# iscsitadm create tpgt 1
# iscsitadm modify tpgt -i 10.10.0.1 1
# iscsitadm modify tpgt -i 10.10.0.2 1
# iscsitadm modify target -p 1 ss1-zstore0

To delete a tpgt that is not correct is very strait forward.

# iscsitadm delete target -p 1 ss1-zstore0
# iscsitadm delete tpgt -A 1

Where 10.20.0.1 and 2 are the target interfaces that should participate in portal group 1 and ss1-zstore0 is the target alias. In some cases you may have to remove the tpgt  all together. The backing store is editable as well as many other SMF properties. To change a backing store value in the SMF we use the svccfg command as follows.

Here is an example of listing all the backing stores and then changing the /dev/zvol/rdsk/sp2/iscsi/lun0 so its on zpool sp1 instead of sp2

# svcadm enable iscsitgt
# svccfg -s iscsitgt listprop | grep backing-store

param_dr-zstore0_0/backing-store                astring  /dev/zvol/rdsk/sp2/iscsi/lun0
param_dr-zstore0_1/backing-store                astring  /dev/zvol/rdsk/sp1/iscsi/lun1

# svccfg -s iscsitgt setprop param_dr-zstore0_0/backing-store=/dev/zvol/rdsk/sp1/iscsi/lun0
# svccfg -s iscsitgt listprop | grep backing-store

param_dr-zstore0_0/backing-store                astring  /dev/zvol/rdsk/sp1/iscsi/lun0
param_dr-zstore0_1/backing-store                astring  /dev/zvol/rdsk/sp1/iscsi/lun1

Changing the backing store value is instrumental if you wish to mount the VMFS LUN’s to provision system validation or online testing. However do not attach the  file system from the active replicated zfs backing store to the ESXi server for validation or testing as it will fail any additional replications once it is modified outside of the active replication stream. You must first create a clone of a chosen snapshot and then modify the backing store to use this new backing store path. This method will present a read/write clone through the iscsitgt service and will have the same iqn names so no reconfiguration would be required to create different time windows into the data stores or reversion to a previous point.

Here is an example of how  this would be accomplished.

# zfs create sp1/iscsi/clones
# zfs clone sp1/iscsi/lun0@10-10-2008-23:45 sp1/iscsi/clones/lun0
# svcadm refresh iscsitgt
# svcadm restart iscsitgt

To change to a different snapshot time you would simply need to destroy or rename  the current clone and replace it with a new or renamed clone of an existing snapshot on the same clone backing store path.

# zfs destroy sp1/iscsi/clones/lun0
# zfs clone sp1/iscsi/lun0@10-11-2008-23:45 sp1/iscsi/clones/lun0
# svccfg -s iscsitgt setprop param_dr-zstore0_0/backing-store=/dev/zvol/rdsk/sp1/iscsi/clones/lun0
# svcadm refresh iscsitgt
# svcadm restart iscsitgt

VMware Software iSCSI configuration

The ESXi iSCSI software configuration is quite strait forward. In this architecture we need to place an interface of the OpenSolaris iSCSI target host on vSwitch1 which is where we defined the iSCSI-Net0 VM kernel network. To do this we create a VM Network portgroup on the same VLAN ID as the iSCSI VM kernel interface.

Here is an example of what this configuration looks like.

DR Net example by Mike La Spina

For more deatail on how to configure the iSCSI VM interfaces see this blog http://blog.laspina.ca/ubiquitous/running_zfs_over_iscsi_as in this case you would not need to define an aggregate since there is only one interface for the iSCSI vSAN.

The final step in the configuration is to define a discovery target on the iSCSI software configuration panel and then rescan the vmhba for new devices.

Hopefully this blog was of interest for you.

Til next time….

Mike

Share
  • Share

Tags: , , , , , , , , ,

Site Contents: © 2008  Mike La Spina

20 Comments

  • Hany Michael says:

    Mike,
    Great article…I hope I will have the time to test this in my lab one day..
    Keep up the good work!

  • Giacomo Marconi says:

    hi! good work.
    I am trying to setup this:
    two freenas iscsi target (cause it boot from usb, and I have all disks free).
    one vm opensolaris as iscsi initiator with a mirrored zpool using the two iscsi target as devices:

    NAME STATE READ WRITE CKSUM
    iSCSI ONLINE 0 0 0
    mirror ONLINE 0 0 0
    c9t3d0p0 ONLINE 0 0 0 first iscsi target
    c9t4d0p0 ONLINE 0 0 0 second iscsi target

    the opensolaris vm is sharing storage via nfs to vmware vsphere cluster.
    At the moment I am working on the failover of the initiator/nfs server and ethernet bonding.

    what do you think about this way ?

    Giacomo

  • Hi,

    That method is fine. NFS is a protocol that also works well when virtualized.

    Regards,

    Mike

  • tranceking says:

    when will you have the GFS snapshot script. Also is the vm in a consistent state when the snapshot is taken?

  • Hi,

    Are you referring to the one listed here http://blog.laspina.ca/ubiquitous/zfs_snapshot_rollup_bash_script

    VM consistency actions like quiescence must be initiated externally from the script. The VMFS volume will be consistant.

    Regards,

    Mike

  • Gregory Durham says:

    Mike,
    Thank you so much for having this blog, it has been incredibly helpful. I am using a solution similar to yours however I have questions about the consistency of the virtual machines. I hope this makes sense, I have two opensolaris machines with a bunch of hard disks, one acts as a iSCSI SAN, and the other is identical other than the hard disk configuration. The only thing being served are VMWare esxi raw disks in physical compatibility mode (no vmware snapshots), which hold either virtual machines(combination of various *nix and windows servers) or data that the particular virtual machine uses, I.E. we have exchange 2007 virtualized and through its iSCSI initiator we are mounting two LUNs one for the database and another for the Logs, all on different arrays of course. Any how we are then snapshotting this data across the SAN network to the other box using snapshot send/recv. In the case the other box fails this box can immediatly serve all of the iSCSI LUNs. The problem, I don’t really know if its a problem…Is when I snapshot a running vm will it come up alive in esxi or do I have to accomplish this in a different way. These snapshots will then be written to tape with bacula.

    Thanks,
    Greg

  • Hi Greg,

    I’m not sure what you mean by “when I snapshot a running vm will it come up alive in esxi” but I assume its about the include memory snapshot option.

    If you snapshot a VM and include the memory image you can revert to that snapshot and it will have the same state the machine was in at the point in time the snapshot was created. There are some exceptions to this action. The uuid of the machine and underlying infrastructure cannot change. e.g. CPU model, CDRom client attributes etc..

    Generally this would be a manual function after you power the VM back up.

    Regards,

    Mike

  • m0ps says:

    Hi Mike! I read articles in your blog and i find it’s very useful for me. I try to implement something like described in this post, but i use COMSTAR’s iSCSI target.
    We have 2 ESX hosts and one shared storage server with OSOL b134 on it, and we want implement backup storage on VM, that placed on one ESX host (that have large local storage). I configure zfs snapshot’s sending/receiving from shared storage server to backup storage VM with bash script provided in this post.
    I have some question: how can I copy (automated) COMSTAR’s iSCSI target configuration from ss1 to ss2, and what steps i may produce after ss1 fail for disaster recovery.
    P.S. sorry for my English…
    Thanks,
    Maxim

  • Mi Maxim,

    You have discovered that COMSTAR is very different.

    Here is what you need to do:

    Define all your host groups, initiators, targets etc. for both ss1 and ss2 on ss1.

    Use the following to export the stmf service config to a file:

    ss1:~# svccfg export -a stmf > stmf.cfg

    Copy the config file to ss2 and perform the following:

    ss2:~# svcadm disable stmf
    ss2:~# svccfg delete stmf
    ss2:~# svccfg import stmf.cfg
    ss2:~# svcadm refresh stmf
    ss2:~# svcadm enable stmf

    Once this is complete you will need to import the LU’s (the last time I worked on this task was on snv_126 so this part may not be required any more):

    ss2:~# stmfadm import-lu /dev/zvol/rdsk/sp1/san/vol0

    And there you go a completely migrated COMSTAR config and LUs.

    Enjoy.

    Regards,

    Mike

  • m0ps says:

    Thanks for your reply Mike!
    I try your solution and it work, but i faced with problem:
    after enabling of stmf and iscsi/target:default on ss2 I can’t continue replication of zvol’s from ss1 to ss2. Script output is “cannot receive: failed to read from stream”
    Also I can’t do svccfg export -a stmf > stmf.cfg from zfsadm user (but svccfg export stmf > stmf.cfg works fine).

  • m0ps says:

    sory… message is:
    cannot receive incremental stream: destination DATA/iSCSI/lun0 has been modified since most recent snapshot

  • Maxim,

    I’m sorry, I missed an important detail, you should only activate (import) the LUNs using a cloned LUN other wise the replication stream will be broken.
    You will need to issue a manual send receive to correct the modified state or possibly redo the entire send by deleting it on the target.
    Locate the last successful snap on the source (ss1) and issue the following.

    ss1:~# pfexec /sbin/zfs send -R sp1/san/volx@time | ssh zfsadm@ss2 pfexec /sbin/zfs receive -F sp1/san/volx

    Regard,

    Mike

  • m0ps says:

    I solved this problem. Just added -F option to second zfs recv instance in bash script. But then there was another problem: can not get a snapshot, due to the fact that zvol is busy. It was solved in the following way: stop stmf and iscsi/target services and rebooting OS. Not exactly a nice way, but I hope that I never do it :) .
    I still could not find solution for problem with exporting stmf configuration with -a option. Problem with user permissions, how can I grant zfsadm user do it?
    Thanks,
    Maxim

  • Maxim,

    If you explore /etc/security/auth_attr you will find all the security descriptions and attributes.

    e.g.
    solaris.smf.read.stmf:::Read STMF Provider Private Data::help=SmfSTMFRead.html

    Here is what you would grant zfsadm:

    ss1:~#usermod -A solaris.smf.read.stmf zfsadm

    Mike

  • jhn says:

    have been using OpenSolaris 5,11 ZFS to create iSCSI LUN for our ESX servers. I am adding another OpenSolaris 5.11 box for replication purposes. Security is not an issue, so I’m using netcat (nc).

    I have tried this on the source server:
    zfs create -V300GB tank/target01

    [I attach target01 to esx server as an iSCSI LUN & create VM in LUN ]

    zfs snapshot tank/target01@snapshot01
    zfs send -R tank/target01@snapshot01 | nc san01 3141

    And, this on the destination server:
    nc -l -p 3141 -vvv | zfs recv -vd tank

    However, the destination LUN does not show up when I rescan the iSCSI host bus adapter.

    What am I doing wrong? How should I be doing this? Thank you.

  • Hi John,

    You have not indicated what iSCSI target your running but I suspect it’s COMSTAR. If that’s the case have a look at this comment and see if it’s what your looking for. http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs/comment-page-1#comment-789

    Regards,

    Mike

  • jhn says:

    Hi Mike, thank you for you response.

    I would guess I’m using COMSTAR from the package inforation, I had just installed all the “scsi” packages from the GUI:

    # pkginfo | grep scsi
    system SUNWiscsidmr Sun iSCSI Data Mover (Root)
    system SUNWiscsidmu Sun iSCSI Data Mover (Usr)
    system SUNWiscsir Sun iSCSI Device Driver (root)
    system SUNWiscsitgtr Sun iSCSI Target (Root)
    system SUNWiscsitgtu Sun iSCSI Target (Usr)
    system SUNWiscsitr Sun iSCSI COMSTAR Port Provider (root)
    system SUNWiscsitu Sun iSCSI COMSTAR Port Provider
    system SUNWiscsiu Sun iSCSI Management Utilities (usr)
    system SUNWmpsvplr Sun MP API library for the scsi_vhci driver (Root)

    But, all I did to get iSCSI working was:

    # zfs set shareiscsi=on tank

    The “LUN’s” just inherit it from the pool:

    # zfs get shareiscsi
    NAME PROPERTY VALUE SOURCE
    tank shareiscsi on local
    tank/target01 shareiscsi on inherited from tank
    tank/target01@snapshot01 shareiscsi on inherited from tank
    rpool shareiscsi off default
    rpool/ROOT shareiscsi off default
    rpool/ROOT/opensolaris shareiscsi off default
    rpool/ROOT/opensolaris@install shareiscsi off default
    rpool/dump shareiscsi off default
    rpool/export shareiscsi off default
    rpool/export/home shareiscsi off default
    rpool/export/home/user shareiscsi off default
    rpool/swap shareiscsi off default

    I looked at the comment you sent and tried running the svccfg export command, but didn’t see anything specific to my tagets:

    # svccfg export -a stmf

    STMF

    Do I need to do something more than just setting shareiscsi?

    Thank you and kind regards.

  • John,

    If you issued shareiscsi=on this action would use the iscsitgtd service which is not COMSTAR and is now deprecated. The process to make it show up on the replicated target is documented in http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs starting at the heading Export SMF iSCSI configuration. Basically you need to load the target system iscsitgtd service manifest properties from the source and then clone the zfs data to a new floder and then change the service manifest properties to point to it.

    Regards,
    Mike

  • [...] tested. My backups are simple san based replicas with ZFS. I run a slightly modified version of this replication script. I modified it to work with zvols and some other small tweaks to the local and remote replication. [...]

Leave a Reply

XHTML: You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>