Backup to remote server

There is a need to replace a slow rdiff-backup with a more nimble solution for incremental backups to a remote server. At first, Rsnapshot was considered, but the reason that he could not do backups to remote servers without crutches was abandoned. Other analogues also did not suit us for one reason or another. We didn’t want to search for something ready in the github open spaces and finish it for ourselves, so it was decided to write a new script from scratch on our own. The main goal is to make a solution for incremental backup to a remote server similar to rdiff, but using rsync hard links.


The script is laid out in our repo on github , we will be happy to receive feedback, tips, commits!

Instructions for applying our solution on the example of CentOS 6.x
')
Preparing server backups.

General setting.

Install rsync and xinetd:
yum -y install rsync xinetd

Add the xinetd service to the load:
chkconfig --add xinetd

Allow rsync
vi /etc/xinetd.d/rsync

Change disable = yes to disable = no
and create the configuration file /etc/rsyncd.conf

And add to it:
pid file = /var/run/rsyncd.pid
log file = /var/log/rsyncd.log

This completes the general setup and proceeds to setting up a backup for a specific server.

Setting the environment for server backup.

We add the user if it is not added yet:

 usernames=backup useradd -g backups $usernames rm -f /home/$usernames/.bash* mkdir /home/$usernames/.ssh /home/$usernames/rsyncbackups chown -R $usernames:backups /home/$usernames chown -R root:root /home/$usernames/.ssh touch /home/$usernames/.ssh/authorized_keys 

The block with the key must be replaced with the generated data id_rsa.pub:
no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa ---your-ssh-key-here--- root@backup.example.com

Add to /etc/rsyncd.conf

 cat << EOF >> /etc/rsyncd.conf [$usernames] comment = backups for $usernames path = /home/$usernames/rsyncbackups use chroot = true uid = root gid = root log file = /var/log/rsyncd/$usernames.log read only = false write only = false hosts allow = 1.2.3.4 hosts deny = * transfer logging = false EOF 

Where:
path = /home/$usernames/rsyncbackups -
log file = /var/log/rsyncd/$usernames.log -
hosts allow = 1.2.3.4 - IP
This completes the configuration of the server part.

Configure the client server.

To change the backup server settings, exclude containers from the backup, local folder (or its absence), it is enough to make changes to the file
rsync-backup.local.conf

Exceptions

For exceptions, there are files with a list of exceptions specified on each new line without a primary slash (exception rules for rsync).

srv/southbridge/etc/rsync-backup.exclude.dist -
srv/southbridge/etc/rsync-backup.exclude.local.example -
srv/southbridge/etc/rsync-backup.exclude.remote.example -

The backup script checks for the presence of the files srv / southbridge / etc / rsync-backup.exclude.local and srv / southbridge / etc / rsync-backup.exclude.remote and, if they exist, adds exceptions for backups. If local backup is canceled, local exceptions are added to remote ones.

Inclusions

A feature of this script is the ability to include certain files or directories from the excluded directory above the hierarchy. For this you need to create files
/srv/southbridge/etc/rsync-backup.include.local
/srv/southbridge/etc/rsync-backup.include.remote
respectively for local and remote inclusions.
If you need to include a specific file, you must specify its path without a primary slash, for example:
var/log/nginx/server.log
If you need to recursively include the directory, then you need to specify the inclusion as follows:
var/log/nginx/**
Every new inclusion from a new line without a primary slash.

When inclusions work, the entire hierarchy of container directories will be backed up, even if an exception of certain directories was added earlier, but directories without files will be backed up.
This is a feature of rsync, unfortunately no other way has been found yet.
Part of the script for work inclusions:

  if [ -f "$LOCAL_INCLUDE" ]; then e "sync include" e "rsync -ax --include=*/ --include-from=$LOCAL_INCLUDE --exclude=* --link-dest=../../Latest $VZ_PRIVATE/$VEID $LOCAL_DIR/$VEID/$WHICH/Processing$DATE" LLOG=`rsync -ax --include=*/ --include-from=$LOCAL_INCLUDE --exclude=* --link-dest=../../Latest $VZ_PRIVATE/$VEID $LOCAL_DIR/$VEID/$WHICH/Processing$DATE 2>&1` fi 

Further work with backups

By default, the script will create backups: 7 days, 4 weeks and 1 month to change this to
/srv/centos-admin.ru/etc/rsync-backup.local.conf you can enter other digits of the following parameters

DAILY = 7
WEEKLY = 4
MONTHLY = 1

Incremental backups are made using hardlinks. Therefore, in order to reduce the amount of disk space occupied by the backup, when adding exceptions, and when cleaning backups from these exceptions, you will need to delete the corresponding directories in each backup folder. Of course, you can write a script to automate this routine, it is in the plans.

In conclusion, what a win in time, we still got:

Comparative test rdiff vs. rsync:

Backup runtime comparison:
1. Primary 5Gb container backup (local and remote)
rdiff-backup - 6 minutes 30 seconds
rsync-backup - 1 minute 34 seconds

2. Secondary backup, there were no changes in the containers
rdiff-backup - 30 seconds
rsync-backup - 2 seconds

3. Secondary backup, a folder with the debian distribution kit (not iso) in 4.7Gb is added to the container
rdiff-backup - 11 minutes 32 seconds
rsync-backup - 2 minutes 15 seconds

And on the charts, the first race is rdiff, the second is rsync