Updated ZFS Replication and Snapshot Rollup Script
Thanks to the efforts of Ryan Kernan we have an updated ZFS replication and snapshot rollup script. Ryan’s OpenIdiana/Solaris/Illumos community contribution improves the script to allow for a more dynamic source to target pool replication and changes the shapshot retention method to a specific number of snapshots rather than a Grandfather Father Son method.
Regards,
Mike
Site Contents: © 2011 Mike La Spina
Subscribe in a reader
Follow me on Twitter
If zfs filesystems with similar name exists in input list the replication fails.
Added @ sign when matching snapshots to allow for similar names.
Regards,
Leif
patch:
*** zfs-replication.sh 2012-02-07 13:06:30.729053500 +0100
— /tmp/zfs-replication.sh 2012-02-07 13:06:09.781167636 +0100
***************
*** 104,110 ****
ssh -n $dhost pfexec zfs create -p $zfspath
ssh -n $dhost pfexec zfs set mountpoint=none $zfspath
! last_snap_shost=$( pfexec zfs list -o name -t snapshot -H | grep $zfspath | tail -1 )
echo $(date) “->” $last_snap_shost Initial replication start. >> replicate.log
pfexec zfs send -v -R $last_snap_shost | ssh $dhost pfexec zfs recv -v -F -d $zfspool
echo $(date) “->” $last_snap_shost Initial replication end. >> replicate.log
— 104,110 —-
ssh -n $dhost pfexec zfs create -p $zfspath
ssh -n $dhost pfexec zfs set mountpoint=none $zfspath
! last_snap_shost=$( pfexec zfs list -o name -t snapshot -H | grep “${zfspath}@” | tail -1 )
echo $(date) “->” $last_snap_shost Initial replication start. >> replicate.log
pfexec zfs send -v -R $last_snap_shost | ssh $dhost pfexec zfs recv -v -F -d $zfspool
echo $(date) “->” $last_snap_shost Initial replication end. >> replicate.log
***************
*** 165,171 ****
if [ "$zfspath" != "" ]
then
! pfexec zfs list -o name -t snapshot | grep $zfspath > $snap_list
while read snaps
do
— 165,171 —-
if [ "$zfspath" != "" ]
then
! pfexec zfs list -o name -t snapshot | grep “${zfspath}@” > $snap_list
while read snaps
do
Hello,
Thank you for putting together such a useful script. I do have 2 questions though.
1. has the above patch been applied to the script that is available for download? if not, how do I apply the patch? is the patch required?
2. What does a sample input file look like?
Thank you.
Allan
Allan,
The patch is applied to the linked script and I have updated the comments to indicate what the input file format should be.
Regards,
Mike
Hi Mike
I want to just replicate/send a snapshot to a different pool on the same system, can i do that with this script, please excuse the ignorance i’m new to ZFS
thanks
Paul
Hi Paul,
This script and it’s associated ssh key pair setup are really intended for an external host over the network. For a local system it is not required and you can simply issue the command locally
For example:
pfexec zfs send somepool/somefilesystem@snapname | pfexec zfs recv sameorotherpool/somefilesystem
will provide the same result without the overhead.
The script will not work unmodified.
You could remove all the ssh portions and hard code the target pool in the script and it would provide the result as well.
Regards,
Mike
Looking through the code, I notice there’s a source $zfspath, but no destination, which means the pool/tank name on both source and destination must be the same?
Would it be better to do something like this?
input file: pool1/zfsstore,host1,pool2/replicationstore,host2
This means you’d need to have a $zfspoolSrc, $zfspathSrc and corresponding $zfspoolDest and $zfspathDest, starting with function parse_rep_list().
And without even more modification, I believe the zfs send would create a $zfspoolDest/$zfspathDest/basename($zfspathSrc), e.g. if you are replicating “pool1/share1/filesystem1″, with destination “pool2/share2/filesystem2″, you’d end up with “pool2/share2/filesystem2/filesystem1″?
Hi Yu-Phing,
That would certainly enable a more dynamic capability for the script and yes it would create the zfs file system but not always the way you want.
This method can create issues if there are clones in the source ZFS files system.
I would recommend keeping the source path intact on the destination e.g SrcPool/SrcName -> DestPool/SrcName.
You also need to be carefull with NFS mounts.
For example if the target host already contained an NFS mount point of /export/vol1 to some portion of the replicated ZFS file system then any source NFS share with the same mount point would conflict.
NFS properties will be inherited as a property of the ZFS file system.
Regards,
Mike
there a typo in the file line 79
$$ZFS should be $ZFS
Corrected.
Thanks Nicolas!
Here’s a diff to get this script working correctly on a FreeBSD 9.0 system. /usr/ports/shells/bash is required. Rudimentary locking is in place as well.
— /tmp/zfs-replication.sh 2012-04-22 09:02:50.000000000 -0500
+++ /root/bin/zfs-replication.sh 2012-05-18 16:06:35.000000000 -0500
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/local/bin/bash
#
# Author: Ryan Kernan
# Date: September 23, 2011
@@ -24,18 +24,23 @@
#
#
#
-
-ZFS=/usr/sbin/zfs
-DATE=/usr/gnu/bin/date
+ZFS=/sbin/zfs
+DATE=/bin/date
SSH=/usr/bin/ssh
-PFEXEC=/usr/bin/pfexec
-export PATH=/usr/gnu/bin:/usr/bin:/usr/sbin:/sbin # Set Paths.
+PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin
rep_list=$1
keep_snaps=7
snap_list=snaplist.lst
+if [ -e /tmp/zfs-rep.lock ];
+then
+ exit
+else
+ touch /tmp/zfs-rep.lock
+fi
+
#######################################################################################
####################################Function###########################################
#######################################################################################
@@ -75,8 +80,8 @@
last_snap_dhost=”"
last_snap_shost=”"
- last_snap_shost=$( $PFEXEC $ZFS list -o name -t snapshot | grep $zfspath | tail -1 )
- last_snap_dhost=$( $SSH -n $dhost $PFEXEC $ZFS list -H -o name -r -t snapshot | grep $zfspath | tail -1 )
+ last_snap_shost=$( $ZFS list -o name -t snapshot | grep $zfspath | tail -1 )
+ last_snap_dhost=$( $SSH -n $dhost $ZFS list -H -o name -r -t snapshot | grep $zfspath | tail -1 )
true
}
@@ -96,7 +101,7 @@
dhost_fs_exists() {
dhost_fs_name=”"
- dhost_fs_name=$($SSH -n $dhost $PFEXEC $ZFS list -o name -H $zfspath | tail -1)
+ dhost_fs_name=$($SSH -n $dhost $ZFS list -o name -H $zfspath | tail -1)
if [ "$dhost_fs_name" = "" ]
then
@@ -119,11 +124,11 @@
dhost_create_fs() {
- $SSH -n $dhost $PFEXEC $ZFS create -p $zfspath
- $SSH -n $dhost $PFEXEC $ZFS set mountpoint=none $zfspath
- last_snap_shost=$( $PFEXEC $ZFS list -o name -t snapshot -H | grep “{$zfspath}\@” | tail -1 )
+ $SSH -n $dhost $ZFS create -p $zfspath
+ $SSH -n $dhost $ZFS set mountpoint=none $zfspath
+ last_snap_shost=$( $ZFS list -o name -t snapshot -H | grep “$zfspath\@” | tail -1 )
echo $($DATE) “->” $last_snap_shost Initial replication start. >> replicate.log
- $PFEXEC $ZFS send -v -R $last_snap_shost | $SSH $dhost $PFEXEC $ZFS recv -v -F -d $zfspool
+ $ZFS send -v -R $last_snap_shost | $SSH $dhost $ZFS recv -v -F -d $zfspool
echo $($DATE) “->” $last_snap_shost Initial replication end. >> replicate.log
}
@@ -141,7 +146,7 @@
snap_date=”$($DATE +%m-%d-%y-%H:%M)”
echo $($DATE) “->” $zfspath@$snap_date Snapshot creation start. >> replicate.log
- $PFEXEC $ZFS snapshot $zfspath@$snap_date
+ $ZFS snapshot $zfspath@$snap_date
echo $($DATE) “->” $zfspath@$snap_date Snapshot creation end. >> replicate.log
}
@@ -160,7 +165,7 @@
incr_repl_fs() {
echo $($DATE) “->” $last_snap_dhost $last_snap_shost Incremental send start. >> replicate.log
- $PFEXEC $ZFS send -I $last_snap_dhost $last_snap_shost | $SSH $dhost $PFEXEC $ZFS recv -d -F $zfspool >> replicate.log
+ $ZFS send -I $last_snap_dhost $last_snap_shost | $SSH $dhost $ZFS recv -d -F $zfspool >> replicate.log
echo $($DATE) “->” $last_snap_dhost $last_snap_shost Incremental send end. >> replicate.log
}
@@ -182,28 +187,24 @@
if [ "$zfspath" != "" ]
then
- $PFEXEC $ZFS list -o name -t snapshot | grep „${zfspath}\@‰ > $snap_list
+ $ZFS list -o name -t snapshot | grep $zfspath\@ > $snap_list
while read snaps
do
- stringpos=0
- let stringpos=$(expr index “$snaps” @)+1
- SnapDate=$( $PFEXEC expr substr $snaps $stringpos 8 )
- let stringpos=$($PFEXEC expr index “$snaps” @)+10
- SnapTime=$( $PFEXEC expr substr $snaps $stringpos 5 )
+ SnapDate=`echo $snaps | cut -d @ -f 2`
SnapDay=”$(echo $SnapDate | cut -d- -f2)”
SnapMonth=”$(echo $SnapDate | cut -d- -f1)”
SnapYear=”$(echo $SnapDate | cut -d- -f3)”
SnapDate=”$SnapMonth-$SnapDay-$SnapYear”
- if [ "$($DATE +%m-%d-%y --date="$keep_snaps days ago")" = "$SnapDate" ]
+ if [ "$($DATE -v-${keep_snaps}d +%m-%d-%y)" = "$SnapDate" ]
then
echo “Destroying $SnapDate snapshot $snaps on $shost” >> replicate.log
- $PFEXEC $ZFS destroy $snaps
+ $ZFS destroy $snaps
echo “Destroying $SnapDate snapshot $snaps on $dhost” >> replicate.log
- $PFEXEC $SSH -n $dhost $PFEXEC $ZFS destroy $snaps
+ $SSH -n $dhost $ZFS destroy $snaps
fi
done > replicate.log # Get the start and stop snaps.
@@ -260,4 +260,7 @@
done < $rep_list
+rm /tmp/zfs-rep.lock
+
exit 0
+
Thanks for your community contribution!