Rotating LVM snapshots for shadow copy

From SambaWiki

Purpose

Use this script to utilize rotating LVM snapshots. The snapshots can then be exported via Samba and the Volume Shadow Copy interface to Windows clients (remember to install the client software) to access the content of the snapshots.

Usage

  • Install the script into /root/bin/smbsnap
  • Create a crontab like this:
0 12 * * 1-5 root /root/bin/smbsnap autosnap all 0
0 7 * * 2-5 root /root/bin/smbsnap autosnap all 1
0 7 * * 1 root /root/bin/smbsnap autosnap all 2
0 8 * * * root /root/bin/smbsnap clean all
3,33 * * * * root /root/bin/smbsnap autoresize all
  • Create /etc/samba/smbsnap.conf like this:
SnapVolumes=('/dev/prod0/source;2000;500;1000;,nouuid')
SnapSets=(2 5 20)
OffDays="Sat Sun"

This is assuming that your LVM volume is /dev/prod0/source and it is formatted with XFS. For other filesystems you should omit the ";,nouuid" part.

  • Add this to some boot script (e.g. /etc/init.d/boot.local on SuSE):
# mount smbsnap snapshots
/root/bin/smbsnap mount all
  • Ensure your configuration is correct in smb.conf. Example:
# This example was verified on Ubuntu 18.04 LTS on 2019-09-05
# Assuming your Samba shares are all on an LVM partition /srv
# and you use the "userhomes" at /srv/samba/userhomes
# and you have a share exampleshare at /srv/samba/shares/exampleshare
# then smbsnap.conf will create your snapshots at /srv/ such as 
# /srv/@GMT-2019.09.05-13.00.00
# in which case you could add the following to your smb.conf configuration:

[global]
vfs objects = shadow_copy2
shadow:mountpoint = /srv
shadow:snapdir = /srv

[userhomes]
path = /srv/samba/userhomes
shadow:snapsharepath = samba/userhomes

[exampleshare]
path = /srv/samba/shares/exampleshare
shadow:snapsharepath = samba/shares/exampleshare

# note that any share where you do not configure "shadow:snapsharepath" will not use shadow copy
  • Restart the smbd process after adding configuration to smb.conf
  • Enjoy

Notes

  • Too many LVM snapshots severly degrade storage performance, the above mentioned setup worked fine for me
  • If your system gets messed up you will have to remove broken snapshots manually with lvremove
  • You don't want your snapshots to overflow (they just get disabled), so set the growth parameter to a reasonable large size corresponding to the maximum data intake of your server

The Script

#!/bin/bash
#
# written by Christian Schwamborn
# bugs and suggestions to:
# christian.schwamborn[you-know-what-comes-here]nswit.de
#
# published under GNU General Public License v.2
# version 1.0.3 (2007-12-13)
#
# Authors: Christian Schwamborn [CS]
#          Schlomo Schapiro [GSS] sschapiro[you-know-what-comes-here]probusiness.de
#
# History:
# 1.0.1 (2006-11-21) CS  initial release
# 1.0.2 (2007-01-08) CS  snapshottime now in GMT
# 1.0.3 (2007-12-13) GSS added support for extra mount options (for XFS)
#
# You are using this scrip at your own risk, the author is not responsible
# for data damage or losses in any way.
#
# What this is:
# You can use this script to create and manage snapshots for the Samba
# VFS module shadow_copy.
#
# How to use:
# The script provides some commanline parameters which are usefull for
# start/stop scrips (i.e. mount and unmount). Other parameters are usefull
# for cronjobs - add this, for a usual snapshot scenario (without trailing #)
# to your crontab:
#
# 0 12 * * 1-5 root /usr/local/sbin/smbsnap autosnap 0 0
# 0 7 * * 2-5 root /usr/local/sbin/smbsnap autosnap 0 1
# 0 7 * * 1 root /usr/local/sbin/smbsnap autosnap 0 2
# 0 8 * * * root /usr/local/sbin/smbsnap clean all
# 3,33 * * * * root /usr/local/sbin/smbsnap autoresize all
#
# This takes snapshots at 7:00 and 12:00 every workday and checks every hour
# if a snapshot needs to be resized.
#
# The script has some flaws:
#   -This script currently works only with LVM2, no EVMS support yet
#   -XFS should be easy to implement, but it isn't yet
#   -You must not use dashes in your volumegroups or logical volumes
#   -Be carefull with the configuration, the parameters are not completely
#    checked right now, as the same for the command line parameters
#   -You have to keep track of the freespace of your volumegroups
#   -Be aware, that if your snapshots grow faster than you assumed, they will
#    become unusable. With the configuration shown above, this script checks
#    every 30 minutes if the snapshots are in the need of a resize. If
#    someone has a better idea how to check the snapshots than periodical,
#    let me know plaese.
#
# This script is written for the bash, other shells might work, it also uses
# some external commands: mount, umount, grep, date, bc, logger, lvcreate,
# lvremove, lvresize
#
# There are currently three variables that have to be configured:
#   -SnapVolumes is an array, every element of that array represents a logical
#    volume that is configured for snapshots. Each element is a comma seperated
#    list, which consists of the logical volume itself (i.e. /dev/GROUP1/foo),
#    the start size of the snapshot (in megabytes), the freespace which should
#    be maintained (in megabytes), the space added, when a snapshot is
#    resized (also in megabytes) and optionally additional mount options required
#    for mounting the snapshot, like ",nouuid" for XFS. Please add the leading ","
#    because this parameter will be appended to "mount -o ro" *verbatim*.
#    The number of an element is used as a reference when calling the script
#   -SnapSets is also an array, currently every element just represents the
#    age (in days) of a snapshot of the specific snapshot-set.
#   -OffDays is a simple string with the none work days.
#
# The script will figure out by itself where to mount the snapshots, but the
# original logical volumes has to be mounted fist.
#
# Copy and adjust the following three variables (without #) to a blank file in
# /etc/samba and name it smbsnap.conf. If you place the configuration file
# elsewhere, make sure to adjust the path below.
#
# SnapVolumes=('/dev/GROUP/foo;2000;500;1000;,nouuid' '/dev/GROUP/bar;3000;1000;2000')
# SnapSets=(2 5 20)
# OffDays="Sat Sun"
#
# NOTE TO USERS OF PREVIOUS VERSION !!
#
# The delimiter changed from , to ; to support adding multiple mount options
#
# please convert your smbsnap.conf with sed -e 's/,/;/g' -i /etc/samba/smbsnap.conf
#
# Sorry for the invonvenience ...
#
###############################################################################

	. /etc/samba/smbsnap.conf

	export LANG=en_US.UTF-8
	export LANGUAGE=en_US:en
	SnapDate=$(date -u +%Y.%m.%d-%H.%M).00

	[ -z "${1}" ] || Command=${1}
	[ -z "${2}" ] || LVolume=${2}
	[ -z "${3}" ] || SnapSet=${3}


	ExtraMountOptions=

	# process a single snapshot
	# arguments: Command
	# needs:     SnapShot, VolumePath, SnapSets, OffDays, FreeSize, ReSize
	# provides:  na.
	# local:     cmd, SnapShotPath, CurrSnapSets, Count, Expire, Parameters, SnapState, CurrSize, FillPercet, CurrFreeSize
	function DoSnap()
	{
		cmd=${1}
		SnapShotPath=${VolumePath}/@GMT-$(echo ${SnapShot##*/} | cut -f3-4 -d\-)

		case ${cmd} in
			# to mount snapshots
			mount)
				[ -d ${SnapShotPath} ] || mkdir ${SnapShotPath} || \
					logger "${0}: ***error*** - unable to create mountpoint for ${SnapShot}"
				if mount | grep -q ${SnapShotPath}; then
					logger "${0}: ***error*** - snapshot ${SnapShot} is allready mounted to ${SnapShotPath}"
				else
					mount ${SnapShot} ${SnapShotPath} -o ro$ExtraMountOptions >/dev/null 2>&1 || \
						logger "${0}: ***error*** - can not mount ${SnapShot} to ${SnapShotPath}"
				fi
			;;

			# to unmount a snapshots
			umount)
				if mount | grep -q ${SnapShotPath}; then
					umount -f -l ${SnapShotPath} >/dev/null 2>&1 || \
						logger "${0}: ***error*** - can not unmount ${SnapShot} mounted to ${SnapShotPath}"
				else
					logger "${0}: ***error*** - snapshot ${SnapShot} is not mounted to ${SnapShotPath}"
				fi
			;;

			# to remove expired snapshots
			clean)
				CurrSnapSet=$(echo ${SnapShot##*/} | cut -f2 -d\-)
				if [ ${CurrSnapSet} -ge 0 ] && [ ${CurrSnapSet} -lt ${#SnapSets[@]} ]; then				
					Expire=$(echo ${SnapSets[${CurrSnapSet}]} | cut -f1 -d,)

					# add off-days, if any, to the expire time; we just count work-days
					declare -i Count=1
					while [ ${Expire} -ge ${Count} ];do
						echo ${OffDays} | grep -q $(date -d "-${Count} day" +%a) && Expire=$((${Expire} + 1))
						Count=$((${Count} + 1))
					done

					# compare date now minus expire-time with the snapshot-date
					if [ $(( $(date +%s) - ${Expire}*24*60*60 - 12*60*60)) -gt \
							$(date -d "$(echo ${SnapShot##*/} | cut -f3 -d\- |  tr \. \-) \
							$(echo ${SnapShot##*/} | cut -f4 -d\- |  tr \. \:)" +%s) ]; then
						# unmount snapshot
						if mount | grep -q ${SnapShotPath}; then
							umount -f ${SnapShotPath} >/dev/null 2>&1 || \
								logger "${0}: ***error*** - can not unmount ${SnapShot} mounted to ${SnapShotPath}"
						fi
						if ! mount | grep -q ${SnapShotPath}; then
							# remove mount-directory
							if [ -d ${SnapShotPath} ]; then
								rmdir ${SnapShotPath} || \
									logger "${0}: ***error*** - unable to remove mountpoint for ${SnapShot}"
							fi
							# finally remove snapshot
							if lvremove -f ${SnapShot} >/dev/null 2>&1;then
								logger "${0}: successfully removed outdated snapshot ${SnapShot}"
							else
								logger "${0}: ***error*** - can not remove logical volume ${SnapShot}"
							fi
						fi
					fi
				else
					logger "${0}: ***error*** - snapshot-set #${CurrSnapSet} of snaphot ${SnapShot} is not configured"
				fi
			;;

			# to check periodical if the snapshots have to be resized
			autoresize)
				Parameters="--options lv_size,snap_percent --noheadings --nosuffix --separator , --unbuffered --units m"
				SnapState=$(lvs ${Parameters} ${SnapShot})
				CurrSize=$(echo ${SnapState} | cut -f1 -d,)
				FillPercet=$(echo ${SnapState} | cut -f2 -d,)
				CurrFreeSize=$(echo "${CurrSize}-${CurrSize}/100*${FillPercet}" | bc)

				if ! [ $(echo "${CurrFreeSize} > ${FreeSize}" | bc) -eq 1 ]; then
					if lvresize -L +${ReSize}M ${SnapShot} >/dev/null 2>&1; then
						logger "${0}: successfully resized snapshot ${SnapShot}"
					else
						logger "${0}: ***error*** - an error occurred while resizing ${SnapShot}"
					fi
				fi
			;;
		esac
	}


	# invoked if all snapshots of a volume are processed
	# arguments: Command
	# needs:     VolumeDevice, VolumePath, SnapSet & functions: DoSnap
	# provides:  SnapShot
	# local:     snapset_tmp, cmd
	function DoAllSnaps()
	{
		cmd=${1}
		[ -z "${SnapSet}" ] || snapset_tmp="${SnapSet}-"
		# checkout if the configured volume exists and is mounted
		if [ -b ${VolumeDevice} ]; then
			if mount | grep -q "${VolumePath} "; then
				# process all snapshots of the volume and, if given, of a specific snapshot-set
				for SnapShot in ${VolumeDevice}-${snapset_tmp}*; do
					if [ ${SnapShot} = "${VolumeDevice}-${snapset_tmp}*" ]; then
						logger "${0}: ***error*** - no backupset #${SnapSet} found for ${VolumeDevice}"
					else
						DoSnap ${cmd}
					fi
				done
			else
				logger "${0}: ***error*** - logical volume ${VolumeDevice} not mounted to ${VolumePath}"
			fi
		else
			logger "${0}: ***error*** - logical volume ${VolumeDevice} does not exist"
		fi
	}


	# creates a new snapshot and mounts it
	# arguments: na.
	# needs:     VolumeDevice, VolumePath, SnapSet, SnapSize, SnapDate & functions: DoAllSnaps, DoSnap
	# provides:  SnapShot
	# local:     na.
	function MakeSnap ()
	{
		case ${SnapSet} in

			[0-9])
				if [ "${Command}" = "autosnap" ]; then DoAllSnaps "clean"; fi
				SnapShot=${VolumeDevice}-${SnapSet}-${SnapDate}
				if lvcreate -L${SnapSize}M -s -n ${SnapShot##*/} ${VolumeDevice} >/dev/null 2>&1; then
					logger "${0}: successfully created new snapshot ${SnapShot}"
				else
					logger "${0}: ***error*** - an error occurred while creating snapshot ${SnapShot}"
				fi
				DoSnap "mount"
			;;

			*)
				echo "usage: ${0} snap|autosnap <LV number | all> <Snap-Set Number>"
			;;
		esac
	}


	# sets some variables and splits the way for certain commands
	# arguments: one object of the array SnapVolumes
	# needs:     Command, & functions: DoAllSnaps MakeSnap
	# provides:  SnapVolume, VolumeDevice, PVGroupName, LVolumeName, VolumePath, SnapSize, FreeSize, ReSize
	# local:     na.
	function SecondChoice ()
	{
		SnapVolume=${1}
		VolumeDevice=$(echo ${SnapVolume} | cut -f1 -d\;)
		PVGroupName=$(echo ${VolumeDevice} | cut -f3 -d/)
		LVolumeName=$(echo ${VolumeDevice} | cut -f4 -d/)
 		VolumePath=$(mount | grep  ^/dev[[:alnum:]/]*${PVGroupName}.${LVolumeName}[\ ] | cut -f3 -d' ')
		SnapSize=$(echo ${SnapVolume} | cut -f2 -d\;)
		FreeSize=$(echo ${SnapVolume} | cut -f3 -d\;)
		ReSize=$(echo ${SnapVolume} | cut -f4 -d\;)
		ExtraMountOptions=$(echo  ${SnapVolume} | cut -f5 -d\;)

		case ${Command} in

			mount|umount|clean|autoresize)
				DoAllSnaps ${Command}
			;;

			snap|autosnap)
				MakeSnap
			;;
		esac
	}


# decides if all configured volumes are processed or just a specific one
# arguments: na.
# needs:     Command, LVolume, SnapVolumes & functions: SecondChoice
# provides:  na.
# local:     snp
case ${Command} in

	mount|umount|snap|clean|autosnap|autoresize)
		case ${LVolume} in

			all)
				for snp in ${SnapVolumes[@]}; do
					SecondChoice ${snp}
				done
			;;

			[0-9])
				if [ ${LVolume} -ge 0 ] && [ ${LVolume} -lt ${#SnapVolumes[@]} ]; then
					SecondChoice ${SnapVolumes[LVolume]}
				else
					logger "${0}: ***error*** - there is no configured logical volume #${LVolume} for snapshots"
				fi
			;;

			*)
			echo "usage: ${0} <command> <LV number | all> [<Snap-Set Number>]"
			;;
		esac
	;;

	*)
		echo "usage: ${0} <command> <LV number | all> [<Snap-Set Number>]"
		echo
		echo "       valid commands are:"
		echo "       mount      - to mount snapshots"
		echo "       umount     - to unmount snapshots"
		echo "       snap       - to make a new snapshot"
		echo "       clean      - to cleanup outdated snapshots"
		echo "       autosnap   - normally used for cronjobs to cleanup"
		echo "                    outdates snapshots an create a new one"
		echo "       autoresize - for a periodical check if snapshots"
		echo "                    needs to be resized"
		echo
		echo "       <LV number> is the number of a logical volume, configured for"
		echo "         snapshots in SnapVolumes, or simply 'all' for all volumes"
		echo "       <Snap-Set Number> is the number of the snapshot-set, configured"
		echo "         in SnapSet. It is optional, except for the commands 'snap' and"
		echo "         'autosnap'"
		echo
	;;
esac