Backup network spheres (samba) in Linux based on Windows Server Backup
We make convenient access to the archives (and create these archives) network ball, for clients working under Windows.
What is good about Windows Server Backup and shadow copies? They are included in the delivery of Windows Server and do not require additional payments (if you do not use cloud archiving), and also do a good job with the tasks assigned to them. For simple usage scenarios - a very worthy solution. And access to shadow copies through the file properties dialog is generally very convenient. Now we will try to do the same for the Linux file server with Samba.
This opportunity gives us the Samba shadow_copy2 module. It must be registered in the network resource section in the smb.conf file:
Unlike the module of the first version, this allows you to place a folder with copies in different places and with different names.
')
Now, if inside the path = /mnt/.share folder we create a subfolder @ GMT-2016.12.25-10.17.52
then nothing will come of it. Add the following settings in the [general] section:
Now in the properties of the network balls, in the section of previous versions, we will see our "copy". Please note that the time is specified in UTC and is converted to a local time zone.
Having a mechanism for accessing copies without a mechanism for creating them is useless. The following script will help us with this (there is an official equivalent ):
For work, you must use LVM ThinVolumes. Compared to regular volumes, their performance is weakly dependent on the number of snapshot (COW reduces performance by 2-3 times while you modify the “fresh” blocks, and if there are 2 or more snapshots, then the work just stops).
The principle of creating backup copies:
Thus, we get a normal incremental backup. If desired, you can adapt for btrfs or ZFS.
Create a snapshot and mount it in a hidden directory with the name of the balls (the folder containing the ball):
Create an archive using rsync to another volume group:
Create a remote archive using rsync:
Create a remote and local archive using rsync:
Delete old copies (leave last N):
Built-in help:
Introduction
What is good about Windows Server Backup and shadow copies? They are included in the delivery of Windows Server and do not require additional payments (if you do not use cloud archiving), and also do a good job with the tasks assigned to them. For simple usage scenarios - a very worthy solution. And access to shadow copies through the file properties dialog is generally very convenient. Now we will try to do the same for the Linux file server with Samba.
Access to previous versions of files
This opportunity gives us the Samba shadow_copy2 module. It must be registered in the network resource section in the smb.conf file:
[share] vfs objects = shadow_copy2 shadow:snapdir = /mnt/.share path = /mnt/share Unlike the module of the first version, this allows you to place a folder with copies in different places and with different names.
')
Now, if inside the path = /mnt/.share folder we create a subfolder @ GMT-2016.12.25-10.17.52
then nothing will come of it. Add the following settings in the [general] section:
wide links = yes # samba unix extensions = no # *nix ( , # ) allow insecure wide links = no # yes unix extensions # wide links , Now in the properties of the network balls, in the section of previous versions, we will see our "copy". Please note that the time is specified in UTC and is converted to a local time zone.
Create archives and snapshot
Having a mechanism for accessing copies without a mechanism for creating them is useless. The following script will help us with this (there is an official equivalent ):
thin_lv_backup.sh
#!/bin/bash # # LVM-ThinVolume BackUp with rsync script set # # (c) 2016 - # Andrew Leshkevich (magicgts@gmail.com) # # # This script set is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by the # Free Software Foundation, either version 2 of the license or, at your # option, any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, see <http://www.gnu.org/licenses/>. # # For a list of supported commands, type 'thin_lv_backup help' # # !!! Please forgive me for bad english !!! # ################################################################################################ ################################################################################################ #Mount the snapshot to the specified mount point, if a snapshot is not active, then activate it # Arguments: # ${1} - Short path to Volume (in VG/LV format) # ${2} - Mount point # ${3} - Optional LMV Volume attribute # Returns: # Return 0 if no errors ################################################################################################ mount_snapshot(){ local SRC=${1} local MNT_TGT=${2} [ "$#" -lt 2 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME SNAPSHOT> <MOUNT POINT> [<VOLUME ATTRIBUTES>]' && return 1 if [ "$#" -eq 2 ]; then local ATTR=$(lvs --noheadings -o lv_attr ${SRC}) else local ATTR=${3} fi findmnt -nf --source /dev/${SRC} >/dev/null 2>&1 && echo "Skip: LV ${SRC} is already mounted!" && return 0 findmnt -nf --target ${MNT_TGT} >/dev/null 2>&1 | grep -v -q ${MNT_TGT} && echo "Skip: the directory ${MNT_TGT} is already a mount point" && return 3 if [ ! -d "${MNT_TGT}" ]; then mkdir -p "${MNT_TGT}" || echo "Error: Creating directory ${MNT_TGT}" || return 4 echo "Info: directory ${MNT_TGT} has been created" fi find ${MNT_TGT} -prune -empty | grep -v -q ${MNT_TGT} && echo "Skip: ${MNT_TGT} directory is not empty" && return 5 [[ ${ATTR} =~ .*a.* ]] || lvchange -ay -K ${SRC} || echo "Error: Volume Activation ${SRC}" || return 6 mount -o ro,nouuid /dev/${SRC} ${MNT_TGT} || echo "Error: Mounting ${MNT_TGT}" || return 7 return 0 } ################################################################################################ # UnMount snaphot, deactivate volume and remove it mount point directory # Arguments: # ${1} - Short path to Volume (in VG/LV format) # Returns: # Return 0 if no errors ################################################################################################ umount_snapshot(){ local SRC=${1} local TGT [ "$#" -ne 1 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME SNAPSHOT>' && return 1 local _TGT=("$( findmnt -nf --source /dev/${SRC} | cut -d ' ' -f 1 )") if [ ! -z "$_TGT" ]; then umount -A /dev/${SRC} || echo "Error: Umounting ${SRC}" || return 2 for TGT in "${_TGT[@]}"; do find ${TGT} -prune -empty | grep -q "${TGT}" && rm --one-file-system -df ${TGT} [ -d "${TGT}" ] && echo "Info: Fail to remove target directory ${TGT}" done fi lvchange -an -K ${SRC} || echo "Error: Volume Deactivation ${SRC}" || return 3 return 0 } ################################################################################################ # Mount all associated snapshots of the volume to its origin mount points # All snapshots must be named on the template: <ORIGIN VOLUME NAME>-GMT-%Y.%m.%d-%H.%M.%S # Arguments: # ${1} - Short path to Origin Volume (in VG/LV format) # ${2} - Optional archive volume group, that used to mount all archive snapshots # Returns: # Return 0 if no errors ################################################################################################ mount_all_snapshot(){ local SRC=${1} local A_VG=${2} local ATTR_S local SNAP [ "$#" -lt 1 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME> [<ARCHIVE VOLUME GROUP>]' && exit 1 IFS=$'/' read -r -a ATTR_S <<< "${SRC}" [ "$#" -eq 2 ] && ATTR_S[0]=${A_VG} local SRC="$( findmnt -nf --source /dev/${SRC} | cut -d ' ' -f 1 )/" local DST_BASE="$( dirname ${SRC} )/.$( basename ${SRC} )/" while IFS='' read -r SNAP; do IFS=$' \t' read -r -a ATTR <<< "${SNAP}" local DST=${ATTR[0]//${ATTR_S[1]}-/} mount_snapshot ${ATTR_S[0]}/${ATTR[0]} ${DST_BASE}@${DST} ${ATTR[1]} || echo "Error: mounting ${ATTR_S[0]}/${ATTR[0]}" done < <( lvs --noheadings -o lv_name,lv_attr -S Origin=${ATTR_S[1]} ${ATTR_S[0]} ) } ################################################################################################ # UnMount and Remove snapshot # Arguments: # ${1} - Short path to Snapshot Volume (in VG/LV format) # Returns: # Return 0 if no errors ################################################################################################ remove_snaphot(){ local TGT=${1} local ATTR_S [ "$#" -ne 1 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME SNAPSHOT>' && return 1 IFS=$'/' read -r -a ATTR_S <<< "${TGT}" [ -z $(lvs --noheadings -o Origin -S lv_name=${ATTR_S[1]} ${ATTR_S[0]}) ] && echo "Error: not a snapshot ${TGT}" && return 2 umount_snapshot ${TGT} || echo "Error: umounting snapshot ${TGT}" || return 3 lvremove -f /dev/${TGT} || echo "Error: removing snapshot ${TGT}" || return 4 return 0 } ################################################################################################ # Create and Mount it to hidden folder on top level with same name as Original mount point # Arguments: # ${1} - Short path to Origin Volume (in VG/LV format) # ${2} - Optional postfix, that replace default postfix GMT-%Y.%m.%d-%H.%M.%S # Returns: # Return 0 if no errors ################################################################################################ create_snaphot(){ local TGT=${1} local ATTR_S [ "$#" -lt 1 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME> [<POSTFIX>]' && exit 1 local DATE=$(date -u +GMT-%Y.%m.%d-%H.%M.%S) [ "$#" -eq 2 ] && DATE="${2}" IFS=$'/' read -r -a ATTR_S <<< "${TGT}" lvcreate -n ${ATTR_S[1]}-${DATE} -s /dev/${TGT} || echo "Error: Creating snapshot of ${TGT}" || return 2 local SRC="$( findmnt -nf --source /dev/${TGT} | cut -d ' ' -f 1 )/" local DST_BASE="$( dirname $SRC )/.$( basename $SRC )/" mount_snapshot ${TGT}-${DATE} ${DST_BASE}@${DATE} || echo "Error: Mounting snapshot ${TGT}-${DATE}" || return 3 } ################################################################################################ # Remove old snaphots and keep last N snapshot # Arguments: # ${1} - Short path to Origin Volume (in VG/LV format) # ${2} - Number of keeping snapshot # Returns: # Return 0 if no errors ################################################################################################ remove_old_snapshot_copy(){ local TGT=${1} local NUM=${2} local SNAP local ATTR_S local ATTR [ "$#" -ne 2 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME> <NUMBER OF KEEP>' && return 1 IFS=$'/' read -r -a ATTR_S <<< "${TGT}" while IFS='' read -r SNAP; do IFS=$' \t' read -r -a ATTR <<< "${SNAP}" local DST=${ATTR[0]//${ATTR_S[1]}-/} remove_snaphot ${ATTR_S[0]}/${ATTR[0]} || echo "Error: removing snapshot ${ATTR_S[0]}/${ATTR[0]}" done < <( (lvs --noheadings -O -lv_name -o lv_name -S Origin=${ATTR_S[1]} ${ATTR_S[0]}) | head -n -${NUM} ) return 0 } ################################################################################################ # Prepare archive operation # Arguments: # ${1} - Short path to Origin Volume (in VG/LV format) # ${2} - Mount point for ${1} # Returns: # Return 0 if no errors ################################################################################################ pre_archive(){ [ "$#" -ne 2 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME> <MOUNT POINT>' && return 1 local VOL_SRC=${1} local MNT_TGT=${2} mkdir -p ${MNT_TGT} mount /dev/${VOL_SRC} ${MNT_TGT} || echo "Error: Mounting ${MNT_TGT}" || return 7 } ################################################################################################ # Post archive operation: unmount target volume, remove its mount point, create its snaphot # Arguments: # ${1} - Short path to Origin Volume (in VG/LV format) # ${2} - Mount point for ${1} # Returns: # Return 0 if no errors ################################################################################################ post_archive(){ [ "$#" -ne 3 ] && echo 'Error: expected <VOLUME GROUP>/<LOGICAL VOLUME> <MOUNT POINT> <VOLUME GROUP>/<LOGICAL VOLUME>' && return 1 local VOL_SRC=${1} local MNT_TGT=${2} local TGT=${3} umount /dev/${VOL_SRC} && rm -rd ${MNT_TGT} && lvcreate -n ${TGT} -s /dev/${VOL_SRC} && return 0 return 1 } ################################################################################################ # Create rsync archive # Arguments: # ${1} - Short path to Origin Volume (in VG/LV format) # ${2} - Name of archive Volume Group # ${3} - Optional connection string in <user name>@<host name> format # ${4} - Optional path to this script on remote machine # ${5} - Optional prefix name for volume name on remote machine (<PREFIX>-<VOLUME NAME>-GMT-%Y.%m.%d-%H.%M.%S) # ${6} - Optional also make local archive # Returns: # Return 0 if no errors ################################################################################################ create_archive(){ local SRC=${1} local TGT=${2} local CONN=${3} local CALL=${4} local PREFIX=${5} local ATTR_S local ATTR_D local RESULTS local RET [ "$#" -lt 2 ] && echo 'Error: expected <ORIG VOLUME GROUP>/<ORIG LOGICAL VOLUME> <DST VOLUME GROUP> [<SSH CONNECT> <SCRIPT CALL> <PREFIX> [<LOCAL COPY?>]]' && return 1 IFS=$'/' read -r -a ATTR_S <<< "${SRC}" IFS=$'/' read -r -a ATTR_D <<< "${TGT}" local SRC="$( findmnt -nf --source /dev/${ATTR_S[0]}/${ATTR_S[1]} | cut -d ' ' -f 1 )/" local DST_BASE="$( dirname $SRC )/.$( basename $SRC )/" create_snaphot ${ATTR_S[0]}/${ATTR_S[1]} archive_orig local DATE=$(date -u +GMT-%Y.%m.%d-%H.%M.%S) if [ "$#" -ge 5 ]; then RESULTS=$(ssh ${CONN} "${CALL} pre_archive ${TGT}/${PREFIX}-${ATTR_S[1]} ${DST_BASE}@archive_dst") RET=$? echo "$RESULTS" [ "${RET}" -ne 0 ] && return ${RET} rsync -aAXx --delete --inplace --no-whole-file ${DST_BASE}@archive_orig/ ${CONN}:${DST_BASE}@archive_dst &&\ RESULTS=$(ssh ${CONN} "${CALL} post_archive ${TGT}/${PREFIX}-${ATTR_S[1]} ${DST_BASE}@archive_dst ${PREFIX}-${ATTR_S[1]}-${DATE}") RET=$? echo "$RESULTS" [ "${RET}" -ne 0 ] && return ${RET} fi if [ "$#" -eq 2 ] || [ "$#" -eq 6 ]; then pre_archive ${TGT}/${ATTR_S[1]} ${DST_BASE}@archive_dst rsync -aAXx --delete ${DST_BASE}@archive_orig/ ${DST_BASE}@archive_dst &&\ post_archive ${TGT}/${ATTR_S[1]} ${DST_BASE}@archive_dst ${ATTR_S[1]}-${DATE} RET=$? [ "${RET}" -ne 0 ] && return ${RET} mount_snapshot ${TGT}/${ATTR_S[1]}-${DATE} ${DST_BASE}@${DATE} else echo 'Error: expected <ORIG VOLUME GROUP>/<ORIG LOGICAL VOLUME> <DST VOLUME GROUP> [<SSH CONNECT> <SCRIPT CALL> <PREFIX> [<LOCAL COPY?>]]>' && return 1 fi remove_snaphot ${ATTR_S[0]}/${ATTR_S[1]}-archive_orig } case ${1} in 'help') [ -z "${2}" ] && echo -e "create - create snapshot and mount it\nmount - mount snapshot\numount unmount snapshot\nmount_all - mount all snapshot\n\ remove - remove snapshot\nremove_old - keep last n snapshot\ncreate_archive - create archive" case ${2} in 'create') echo 'thin_lv_backup.sh create <VOLUME GROUP>/<LOGICAL VOLUME> [<POSTFIX>]' ;; 'mount') echo 'thin_lv_backup.sh mount <VOLUME GROUP>/<LOGICAL VOLUME SNAPSHOT> <MOUNT POINT> [<VOLUME ATTRIBUTES>]' ;; 'umount') echo 'thin_lv_backup.sh umount <VOLUME GROUP>/<LOGICAL VOLUME SNAPSHOT>' ;; 'mount_all') echo 'thin_lv_backup.sh mount_all <VOLUME GROUP>/<LOGICAL VOLUME> [<ARCHIVE VOLUME GROUP>]' ;; 'remove') echo 'thin_lv_backup.sh remove <VOLUME GROUP>/<LOGICAL VOLUME SNAPSHOT>' ;; 'remove_old') echo 'thin_lv_backup.sh remove_old <VOLUME GROUP>/<LOGICAL VOLUME> <NUMBER OF KEEP>' ;; 'create_archive') echo 'thin_lv_backup.sh create_archive <ORIG VOLUME GROUP>/<ORIG LOGICAL VOLUME> <DST VOLUME GROUP> [<SSH CONNECT> <SCRIPT CALL> <PREFIX> [<LOCAL COPY?>]]' ;; esac ;; 'create') create_snaphot $2 $3 ;; 'mount') mount_snapshot $2 $3 $4 ;; 'umount') umount_snapshot $2 ;; 'mount_all') mount_all_snapshot $2 $3 ;; 'remove') remove_snaphot $2 ;; 'remove_old') remove_old_snapshot_copy $2 $3 ;; 'create_archive') create_archive $2 $3 $4 $5 $6 $7 ;; 'pre_archive') pre_archive $2 $3 ;; 'post_archive') post_archive $2 $3 $4 ;; esac disclaimer
The script is not perfect (as long as I fill my hand) and relies on your good faith (there is no check on the correctness of the arguments, only on the number). However, I tried to make it as safe as possible for the data (it does not delete volumes, only snapshot, only deletes empty directories). Your suggestions for improvement and correction are welcome.
For work, you must use LVM ThinVolumes. Compared to regular volumes, their performance is weakly dependent on the number of snapshot (COW reduces performance by 2-3 times while you modify the “fresh” blocks, and if there are 2 or more snapshots, then the work just stops).
The principle of creating backup copies:
- Create a snapshot of the source volume and mount it
- Mount destination volume
- Copy using rsync
- Unmount the destination volume and make it snaphot
- Unmount the snapshot source and remove it
- For local archiving, mount the last snapshot of the archive volume
Thus, we get a normal incremental backup. If desired, you can adapt for btrfs or ZFS.
Using
Create a snapshot and mount it in a hidden directory with the name of the balls (the folder containing the ball):
/usr/local/bin/thin_lv_backup.sh create vg_system/share Create an archive using rsync to another volume group:
/usr/local/bin/thin_lv_backup.sh create_archive vg_system/share vg_archive Create a remote archive using rsync:
/usr/local/bin/thin_lv_backup.sh create_archive vg_system/share vg_archive archive@archive.localdomain 'sudo /usr/local/bin/thin_lv_backup.sh' srv1 Create a remote and local archive using rsync:
/usr/local/bin/thin_lv_backup.sh create_archive vg_system/share vg_archive archive@archive.localdomain 'sudo /usr/local/bin/thin_lv_backup.sh' srv1 true Delete old copies (leave last N):
/usr/local/bin/thin_lv_backup.sh remove_old vg_system/share 5 Built-in help:
/usr/local/bin/thin_lv_backup.sh help /usr/local/bin/thin_lv_backup.sh help mount_all Source: https://habr.com/ru/post/318388/
All Articles