Using hardlinks (hardlink) for Synology DSM incremental backup

The DSM system is quite convenient and, by default, modules are installed in the system that cover 95% of the needs of an ordinary (and not so much) user, which is called “out of the box”.

There is also a built-in backup system: Backup & Replication. Simple, clear and reliable. As a network destination, it can use either a similar Synology or rsync server.
')
Unfortunately, this system does not know how to make incremental backup. The most primitive way to get around this is to set up a separate backup for each day of the week. Thus, we will have 7 folders with backups, but the obvious disadvantage is the storage of a full copy in each folder - the volume may be such that not everyone will allow himself such a repository.

So - let's configure a full incremental backup.

First we need a server with a disk sufficient to store archives of important data. Where exactly it will be installed - it depends on the capabilities of each. For example, it can be a virtual hosting or an old system or a virtual machine on a work computer. You can argue about the reliability of the backup server, but even its very presence is a huge plus to the safety of your personal data. Again, if you enable mail notifications in DSM, you will know daily if there were any errors during the backup and you will quickly bring back the “fallen” server for storing backup copies.

The way that the server will communicate with Synology (or vice versa) everyone will choose for himself - it can be a VPN, a local area network or the usual Internet. I prefer VPN, but the description of the connection settings is beyond the scope of this article (there may be nuances and I will describe this later).

Now the size of the disk on which the archives will be stored.

The contents of the home storage can be confidently divided into: Cinema, Music, Photos, and regular user documents.
And if in case of loss of the data of the Cinema it is possible and not difficult to download again, the Music is very likely too, then you cannot restore the Photos and personal files anymore. The amount of such data from my personal experience does not exceed 50-100GB. For example, we will make a daily and weekly backup. Take the maximum: 100GB + 100GB - we need a 200GB drive. I think? Nowadays it is a rarity, but 500GB is easy to find!

Next, I will talk about the example of a virtual machine, but the basic idea applies to any Linux-based system (possibly FreeBSD).

So, we install the Ubuntu server, and activate SSH and rsync on it:
The file system must have hardlink support. Therefore, we leave the default ext-4.

In principle, rsync can also be run on Windows, but we need support for hardlinks, and I'm not sure that rsync can work correctly with Windows Junction.

I will clarify the security model on the server - it will be quite simple:

- separate user for login (for Synology)
- Restriction on the folder for the IP address at the rsync level
- verification of users at the rsync level - is absent, although it can be easily configured
- if the server is located in unreliable networks on it, it is recommended to enable the firewall and restrict access by IP.

We’ll start the setup - we will create a folder for storing archives and give everyone the rights to it:

$sudo mkdir -p /bakstorage/syno $sudo chmod 777 /bakstorage/syno 

On the server, we need to create an ordinary user, under which Synology will add backup copies:

 $sudo adduser backup 

We rule rsync config:

 cat /etc/rsyncd.conf #  : motd file=/etc/motd log file=/var/log/rsyncd #    [syno] # IP address  Synology hosts allow = 192.168.xx # ,   comment = Syno archive #   ,          path = /bakstorage/syno #     use chroot = yes lock file = /var/lock/rsyncd read only = no list = yes uid = nobody gid = nogroup strict modes = yes ignore errors = no ignore nonreadable = yes transfer logging = no timeout = 600 refuse options = checksum dry-run dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz 

And put it in autoload:

 $sudo update-rc.d rsync defaults 

Reboot, check that everything is up.

Now we set up archiving in DSM. Create Network Backup Destination:



Create a “Data Backup Task”, then create a task for daily backup. Mark the folders you need to save:



Mark applications (optional):



And most importantly - we make it daily (the last option):



We exit to the main application window, select the created task and click “Backup now” - the first copy will take a lot of time. Subsequent will be performed much faster - only new or modified files will be transferred.

After copying is set up and working (you can watch it for several days) - proceed to the next section and add incremental copying. To do this, we use the excellent rsync property:

If the original file was changed, and the destination file is hardlink, then when copying, the hardlink is “broken” and the destination file is replaced with the modified version.

Based on the above rsync property, all we need is to make a copy of the folder with the archive after the backup, with the command: cp -al

This will create a complete copy, but all the files inside will be hardlink, so no additional disk space will be occupied. Well, except that under the tree directories and subdirectories. Let's call it snapshot creation. By the way: for a folder of 100GB in size, creating a snapshot takes no more than a minute.

We can execute this command at any time, after the intended completion of the backup or, for example, an hour before it.
Thus, if the backup starts at 3:00 and lasts 6 hours (a very unfavorable scenario - usually all changes are copied an order of magnitude faster), then snapshot will be done at 10 am - so that we can make sure that everything worked as it should during the day.

The next step is to store N snapshots and automatically delete the oldest ones. By the way, when implementing such a deletion, folders that are older than N days are usually deleted. But if archiving has not been done for a long time (for various reasons), it is likely that all folders with archives will be deleted as obsolete, and there will be no new ones! Therefore, we are learning this.

Below are the scripts that will do all this work for us:

 sudo cat /root/rotate_daily.sh #!/bin/bash #   echo -n "Started: " date # ,   ,   ,     if [[ /bakstorage/syno/daily -nt /bakstorage/syno/daily_full ]]; then #    - «»   /root/rotate_folders.sh /bakstorage/syno/daily_full 3 #     echo "Copying current backup..." cp -al /bakstorage/syno/daily /bakstorage/syno/daily_full else #  -     —    echo "No today backup found!" fi 

 sudo cat /root/rotate_folders.sh #!/bin/bash # ver 1.0 path=$1 hold=$2 if [ -z $2 ]; then cat <<EOT Rotate folders script. Usage: $0 <path_to_folder_prefix> <folders_to_hold> Warning! Folders deleted and renamed by mask: path_to_folder_prefix* so please be sure that no any folders in the path Example: $0 /var/backup/daily 2 The result will be: /var/backup/daily -> /var/backup/daily.1 /var/backup/daily.1 -> /var/backup/daily.2 /var/backup/daily.2 -> Deleted EOT exit 1 fi num=$(ls -d $path* | wc -l) let "num=num-hold" if [ $num -gt 0 ]; then echo "ROTATE_FOLDERS: Found to delete: $num" del=$(ls -d $path* | sort | tail -n $num) echo "ROTATE_FOLDERS: Deleting folder(s):" echo "$del..." rm -r $del else echo "ROTATE_FOLDERS: Nothing to delete." fi # rename let "start=$hold-1" for i in $(seq $start -1 0); do let "y=i+1" if [ $i -eq 0 ]; then echo "ROTATE_FOLDERS: Renaming folders $path to $path.1 ..." mv "$path" "$path.1" else echo "ROTATE_FOLDERS: Renaming folders $path.$i to $path.$y ..." mv "$path.$i" "$path.$y" fi done 

Do not forget to make both scripts executable:

 chmod 750 /root/rotate_folders.sh chmod 750 /root/rotate_daily.sh 

Add to cron:

 $sudo crontab -e 0 10 * * * /root/rotate_daily.sh >>/var/log/rotate_daily.log 2>&1 

Note that all output will be recorded in the file: /var/log/rotate_daily.log

Actually, with daily copying everything!

If we want to add a weekly backup, you can go two ways:

1. Set it up by analogy and in parallel with the daily. This way you will have a completely autonomous set of folders weekly, weekly.1 weekly.2, etc. which will not intersect with daily folders. But in this case it will take up as much extra space on the disk as the daily archives.

2. Modify the script and once a week create an additional snapshot of the daily folder in the weekly folder, and then also rename the weekly folder to weekly.1, weekly.2, etc. In this case, only modified files will usually take up additional space very few, but if any file is corrupted, it will be corrupted in all folders at once!

By the way - the above technique can be applied not only to backups in Synology DSM, but to any system that supports rsync and hardlink.

Successful setting!