Backup for standalone * NIX servers. Emulating TimeMachine

I think none of those present need to explain the importance of backup.
The problem is that of the dozens of ready-made solutions, none of them really meet my requirements for a standalone * NIX server on a collocation.
What did you want from backup?
1) daily full backup of all data. No incremental-bucks.
2) the fastest possible recovery of a single file. Archivers (tar / gzip / bzip2 / rar) disappear
3) fast monitoring “who exactly filled 156GB server yesterday? !!!”
4) you want to keep backup copies as long as possible, how much free space is available on the disks.
5) I do not want to worry about manually deleting old copies if the disk space has run out.
In a nutshell, I wanted to implement the MAC OS TimeMachine functionality on a Linux server.
And I started writing a script.
You probably already guessed that at the heart of TimeMachine is a combination of cp -al + rsync
On the fingers, it looks like this:
cp -al _backup _backup
will copy hardlinks from yesterday’s files to today’s. Such a copy almost does not occupy real disk space. Then it starts
rsync -a --del _backup
it will scan the differences between the copy of yesterday's backup and, if it finds differences, will remove the hardlink and record the new version of the file; create / copy new files and directories; will delete non-existing more files. In this case, all the files in yesterday's backup will remain in place.
It is clear that in order to make a normal working script backup of these two teams is not enough. I decided to write a script in 2 files - one standard for all servers with a set of functions, the second one is short, individual for each server, which is launched directly through cron.

So, here are the scripts: ( can be downloaded from here )
backup_functions.sh

 #!/bin/sh export LC_ALL=en_US.utf8 DIR_PATTERN='20..-..-..' CURR_DATE=`date +%F` #CURR_DATE=`date +%F_%R` RESERVE_G=5 BACKUP_MAIN_DIR='/backup' BACKUP_TMP_DIR='/tmp' BACKUP_DELTA=$BACKUP_TMP_DIR/backup.delta BACKUP_ERRORS=$BACKUP_TMP_DIR/backup.err BACKUP_REPORT=$BACKUP_TMP_DIR/backup.report BACKUP_LOG_FACILITY='user.notice' BACKUP_EXPIRES_DAYS=0 VERIFY_BACKUP_MOUNTED='no' PID_FILE='/var/run/backup.pid' BACKUP_MYSQL_DIR=$BACKUP_TMP_DIR/mysql_dump MYSQL_DATA_DIR='/var/lib/mysql' [ -z "`which rsync`" ] && { echo "RSYNC is not installed! backup will not work!"; exit; } [ -n "`which ionice`" ] && IONICE_CMD='ionice -c2 -n6' touch /etc/default/backup_exclude rm $BACKUP_DELTA $BACKUP_ERRORS $BACKUP_REPORT 1>/dev/null 2>/dev/null verify_backup_mounted() { mount -a [ -d "$BACKUP_MAIN_DIR" ] || { echo "BACKUP main directory does not exist!"; exit; } str=`df "$BACKUP_MAIN_DIR" | tail -1 | grep ' /$'` [ "$str" ] && { echo 'BACKUP partition is not mounted!!!!!!!!'; exit; } return 0 } prepare_for_backup() { if [ -s "$PID_FILE" ] && [ `cat "$PID_FILE"` -ne $PPID ] then if [ "`ps ax | awk '{print $1;}' | grep -f \"$PID_FILE\"`" ] then echo -n "Previous BACKUP script is still running. PID = "; cat "$PID_FILE"; exit else logger -t BACKUP -p $BACKUP_LOG_FACILITY "Previous BACKUP ended unexpectly" fi fi rm "$PID_FILE" 1>/dev/null 2>/dev/null echo $PPID > "$PID_FILE" old_dir=`pwd` cd "$BACKUP_MAIN_DIR" || { echo "BACKUP main directory does not exist!"; exit; } VERIFY_BACKUP_MOUNTED=`echo "$VERIFY_BACKUP_MOUNTED" | tr 'AZ' 'a-z'` [ "$VERIFY_BACKUP_MOUNTED" = "yes" ] && verify_backup_mounted reserve_k=$(($RESERVE_G * 1048576)) mkdir -p $BACKUP_TMP_DIR 1>/dev/null 2>/dev/null dirs_list=`ls | grep $DIR_PATTERN | sort` if [ -n "$dirs_list" ] then while { free_k=`df -k .|grep -v Filesystem| sed -e "s/.\+ \([0-9]\+\) .\+/\1/"` dirs_list=`ls | grep $DIR_PATTERN | sort` free_pre=$free_k [ $free_pre -lt $reserve_k ] ; } do dir_oldest=`echo $dirs_list | tr " " "\n" | head -1` [ -d $dir_oldest ] && { logger -t BACKUP -p $BACKUP_LOG_FACILITY "Deleting old backup in $BACKUP_MAIN_DIR/$dir_oldest" ; rm -rf $dir_oldest; } done fi [ "$VERIFY_BACKUP_MOUNTED" = "yes" ] && verify_backup_mounted last_date=`ls | grep $DIR_PATTERN | sort | tail -1` if [ -n "$last_date" -a \( "$CURR_DATE" != "$last_date" \) ] then logger -t BACKUP -p $BACKUP_LOG_FACILITY "Preparing. Copying $BACKUP_MAIN_DIR/$last_date -> $BACKUP_MAIN_DIR/$CURR_DATE" mkdir $CURR_DATE 1>/dev/null 2>/dev/null $IONICE_CMD cp -al "$last_date"/* $CURR_DATE 1>/dev/null 2>/dev/null rm -rf $CURR_DATE/_delta 1>/dev/null 2>/dev/null fi mkdir $CURR_DATE/_delta 1>/dev/null 2>/dev/null if [ $BACKUP_EXPIRES_DAYS -gt 0 ] then for expired_dir in `find "$BACKUP_MAIN_DIR" -maxdepth 1 -mtime +$BACKUP_EXPIRES_DAYS -type d | grep "$DIR_PATTERN"` do logger -t BACKUP -p $BACKUP_LOG_FACILITY "Deleting expired backup $expired_dir" ; rm -rf $expired_dir; done fi cd $old_dir return 0 } make_backup() { while [ -n "$1" ] do [ "$VERIFY_BACKUP_MOUNTED" = "yes" ] && verify_backup_mounted src=$1 full_src=`echo $PWD/$1 | sed -e 's://:/:g'` dst=`echo $BACKUP_MAIN_DIR/$CURR_DATE/$src | sed -e "s/\/\w\+$//"` mkdir -p $dst 1>/dev/null 2>/dev/null logger -t BACKUP -p $BACKUP_LOG_FACILITY "$full_src started" $IONICE_CMD rsync -axW8 --del --exclude-from=/etc/default/backup_exclude $src $dst 2>>$BACKUP_ERRORS sync shift done return 0 } make_backup_with_delta() { while [ -n "$1" ] do [ "$VERIFY_BACKUP_MOUNTED" = "yes" ] && verify_backup_mounted src=$1 full_src=`echo $PWD/$1 | sed -e 's://:/:g'` dst=`echo $BACKUP_MAIN_DIR/$CURR_DATE/$src | sed -e "s/\/\w\+$//"` mkdir -p $dst 1>/dev/null 2>/dev/null rm $BACKUP_DELTA 1>/dev/null 2>/dev/null logger -t BACKUP -p $BACKUP_LOG_FACILITY "$full_src (with delta) started" $IONICE_CMD rsync -axW8i --del $src $dst --exclude-from=/etc/default/backup_exclude 2>>$BACKUP_ERRORS | grep "^>f" | cut -d ' ' -f 2- 1>$BACKUP_DELTA old_dir=`pwd` cd $BACKUP_MAIN_DIR/$CURR_DATE dst=`echo $src | sed -e "s/\w\+$//"` xargs -a $BACKUP_DELTA -r -n5 -d '\n' -I '{}' echo $dst{} | xargs -r -n10 -d '\n' cp -ul --parents -t _delta rm $BACKUP_DELTA 1>/dev/null 2>/dev/null cd $old_dir sync shift done return 0 } send_email_report() { if [ -s $BACKUP_ERRORS ] then logger -t BACKUP -p $BACKUP_LOG_FACILITY "Sending email report" echo 'Content-type: text/plain; charset=utf-8' >> $BACKUP_REPORT echo 'Content-Transfer-Encoding: 8bit' >> $BACKUP_REPORT echo 'From: root@'`hostname --fqdn` >> $BACKUP_REPORT echo 'To: root' >> $BACKUP_REPORT echo 'Date:' `date` >> $BACKUP_REPORT echo -e 'Subject: Cron <root@'`hostname --fqdn`'> BACKUP\n\n' >> $BACKUP_REPORT cat $BACKUP_ERRORS >> $BACKUP_REPORT cat $BACKUP_REPORT | sendmail root fi rm $BACKUP_DELTA $BACKUP_ERRORS $BACKUP_REPORT $PID_FILE 1>/dev/null 2>/dev/null logger -t BACKUP -p $BACKUP_LOG_FACILITY "Finished" return 0 } make_mysql_backup() { MYSQL_DATA_DIR='/var/lib/mysql' mkdir $BACKUP_MAIN_DIR/$CURR_DATE/MySQL 1>/dev/null 2>/dev/null rm -rf $BACKUP_MAIN_DIR/$CURR_DATE/MySQL/* 1>/dev/null 2>/dev/null rm -rf $BACKUP_MYSQL_DIR 1>/dev/null 2>/dev/null mkdir -p $BACKUP_MYSQL_DIR 1>/dev/null 2>/dev/null cd $MYSQL_DATA_DIR logger -t BACKUP -p $BACKUP_LOG_FACILITY "MySQL started" for db_dir in `ls -p | grep '/' | tr -d '/'` do cd $MYSQL_DATA_DIR/$db_dir db_name=`echo $db_dir | sed -e 's/@003d/=/g' -e 's/@002d/-/g'` logger -t BACKUP -p $BACKUP_LOG_FACILITY "MySQL database '$db_name' started" for table in `ls | grep '.frm' | sed -e 's/\.frm//' -e 's/ /:::/g'` do table=`echo $table | sed -e 's/:::/ /g' -e 's/@003d/=/g' -e 's/@002d/-/g'` mysqldump $db_name "$table" --skip-lock-tables -Q -u $mysql_user -p$mysql_pass > "$BACKUP_MYSQL_DIR/$table.sql" done cd $BACKUP_MYSQL_DIR tar czf $BACKUP_MAIN_DIR/$CURR_DATE/MySQL/$db_name.tar.gz * 1>/dev/null 2>/dev/null rm $BACKUP_MYSQL_DIR/* 1>/dev/null 2>/dev/null done return 0 } 

and backup.sh :

 #!/bin/sh . /usr/local/sbin/backup_functions.sh BACKUP_EXPIRES_DAYS=365 #  30 RESERVE_G=30 BACKUP_MAIN_DIR='/backup' VERIFY_BACKUP_MOUNTED='yes' prepare_for_backup cd / make_backup etc boot home root opt srv usr/local make_backup_with_delta var/spool var/lib var/www BACKUP_MAIN_DIR='/backup' mysql_user='root' mysql_pass='jndey7hdFdfii7HN6ygdrarUh' make_mysql_backup send_email_report 

It is assumed that backup_functions.sh is located in / usr / local / sbin . If in another directory, correct the corresponding line in backup.sh
As you can see, the script abundantly uses global variables. for example
RESERVE_G - the reserve (in gigabytes) that the script will provide by deleting the oldest copies before starting today's backup
BACKUP_EXPIRES_DAYS - after how many days to delete the old backup, even if disk space is still there. if BACKUP_EXPIRES_DAYS == 0 then this is not done.
BACKUP_MAIN_DIR - directory in which the backup is written. Inside there are folders like “2011-11-15” where backup is written.
VERIFY_BACKUP_MOUNTED - check or not check whether a separate partition of the disk is mounted under the backup. It should be understood that if the selected partition was unmounted for some reason, then without this check, the backup will be a heavy burden on rootfs or another unsuitable disk partition. However, there are situations when a backup is explicitly and specifically written on rootfs or another non-specialized section. Then you need to specify
VERIFY_BACKUP_MOUNTED = "no"
a short backup.sh script, which can already be run directly by a cron, calls several self-made functions, namely:
prepare_for_backup - the function prepares everything before copying directly. Roughly speaking, it does cp -al + any checks whether the free disk space has run out.
make_backup (with parameters) - the function actually makes a copy
make_backup_with_delta (with the same parameters) - the function makes a copy and additionally adds all new files to the directory 2011-11-15 / _delta with hardlinks. Looking into this directory you can quickly and easily find out who uploaded 156GB of data to the server yesterday.
send_email_report - the function sends the e-mail to the admin with errors (if any) during the backup.
In addition to backing up files, I also wrote the backup function for MySQL in the appendix.
make_mysql_backup - the function is called without parameters; it makes a public mysqldump (i.e., one table - one .sql file), then all the .sql files of each database are packaged into db_name.tar.gz
')
Sorry for the sheet. I hope someone will come in handy.

UPDATE: if you want to make backups more often than once a day - in the first lines of backup_functions.sh replace the line
CURR_DATE = `date +% F`
on
CURR_DATE = `date +% F_% R`

If rsync is not installed, the script is aborted with the corresponding message.
If ionice is set, then rsync will be launched with a lower io priority (parameters can be changed in the
[-n "` which ionice` "] && IONICE_CMD = 'ionice -c2 -n6'