Here we will look at how to make a differential out-of-box backup system (well, almost), using the minimum number of external modules, in the best UNIX-way traditions.
We will use 7za.exe \ 7z, as well as the UNIX-like utility pdate.exe, so that over time we will work as well as in the tube * NIX, and the replacement for bash will be the “common” BAT. Background and details - under the cut.
Prehistory
It all started with a story about an inattentive and fearless user opening incoming files via personal mail by opening the .exe on the work computer. Unfortunately, this turned out to be not a mediocre Malvar, but a completely ordinary encoder. But, in addition to the user's files, which, of course, cannot be recovered, one tiny but important ball, access to which everyone had, was affected.
Looking at this encrypted indecency, I gratefully remembered that every day I have a backup of this (and not only this) balls built-in Windows Server 2003 SP2 x64. But, having looked through this backup, I realized that in terms of backup by means of Windows itself, not everything is so rosy. First, the full backup was not available, which means it is unlikely to restore cold-data (files that change very rarely). Secondly, the recovery from the created incremental backup turned out to be a nontrivial task - for each step it was possible to recover only the data that was changed, and nothing more. It turns out that to restore at least all the changed data (since the full backup was lost), we would have to go through all the backups in turn - not exactly what I expected from the incremental backup in this case.
')
Some of you can say - it was necessary to check the performance of the backup, and yes, the way it is. But the one of you who works in the trade, can understand where the admin time can go - yes, they are the most, online ticket offices.
Thinking deeply, I remembered my first acquaintance with the fsbackup incremental copying system by Maxim Chirkov
www.opennet.ru/dev/fsbackup - flexibility, simplicity, at the same time, plenty of opportunities and an open archive storage format (tar). It is a pity that the system is designed for * NIX / Linux. Google also did not answer my question about a similar system under Windows. The most useful that I have found is a
short guide of the antip0d
habrovchanin and
an example of a script for backup. I used the material from the last link for my script.
We assemble the system
First of all, download the latest
stable version . At the time of writing it is 16.04. Our backup will be stored in the 7z archive: support for multithreading, encrypted / multi-volume archives, and the extraction speed from 7z is higher than the packaging speed by 10-20 times!
UPD: Thanks to the Taciturn
hacker for the
fix - you can also use 7z.exe already installed on your system. I did not reveal any functional differences between 7z and 7za.
We are interested:
7za.exe is a standalone version of 7-Zip.
7za.dll - library for working with archives 7z
7zxa.dll is a library for unpacking 7z archives.
For 64-bit OS use the same files from the x64 directory.
Unfortunately, the link from the material I used to the pdate utility does not lead anywhere, the only version I found
pdate v1.1 build 2007.12.06
© 2005-2007 Pavel Malakhov 24pm@mail.ru
The link from the built-in manual pdate leads there too, namely, to nowhere.
pm4u.opennet.ru/mysoft/pdate.htm
Fortunately, on the same
resource there is a short article on this program, in the same place it can be
downloaded .
I used the following directory structure:
D: \ winfsbackup - the root directory of the script and related files
D: \ winfsbackup \ 7z - libraries and executable file 7za
D: \ winfsbackup \ backup -
backup storage location (can be reassigned by editing variables, like any other used files)
D: \ winfsbackup \ lists - lists of files to include and exclude. I will tell about them later
D: \ winfsbackup \ log - logs
D: \ winfsbackup \ pdateD: \ winfsbackup \ tmp - sets the working directory for the temporary base archive
D: \ winfsbackup \ winfsbackup.bat - the script itself.
Work logic
After processing the variables, the script looks at the block: Main, where the logic of the backup operation is indicated - in which case the new backup should be executed, and in which case - the existing base archive should be updated. By default, a new archive is created at the beginning of the month, and all files from the backup directory are moved to \ backup \ old, or if the base archive does not exist.
Already at the time of writing this article, I realized that it was necessary to add the ability to update the basic archive - a simplified version of “full backup once a month + differential backups to it” should be used for file exchangers up to ~ 250 GB in size. For my file hosting service of 550 GB with a predominance of small files, the backup speed was unsatisfactory (almost 55 hours). It is fair to say that this cannot serve as a reliable measure of performance - during the backup process it turned out that some files were inaccessible (hello chkdsk), and the backup was added to the remote server section, which was also occupied by disk I / O operations.
:Main REM , \ , . REM - IF NOT EXIST %baseArch% GOTO BaseArchive REM + t IF %dm% EQU 1 GOTO BaseArchive ELSE GOTO UpdateArchive REM 1 REM IF %dm% EQU 1 GOTO UpdateBase ELSE GOTO UpdateArchive REM (2, 19, 36 ) REM IF NOT %wn%.%dw% EQU 02.5 GOTO UpdateArchive REM IF NOT %wn%.%dw% EQU 19.5 GOTO UpdateArchive REM IF NOT %wn%.%dw% EQU 36.5 GOTO UpdateArchive REM , REM IF %dw% EQU 6 (GOTO UpdateBase) ELSE (GOTO UpdateArchive) REM , . REM ECHO Warning! No one condition matching, check :Main block of script >> %Log% GOTO End
The first 2 conditions are enabled by default, and describe a full backup once a month + differential backups. The remaining conditions are alternative, that is, when you turn on one, you need to turn off another, or change the logic of work to your taste.
Variables
dm, dw, wn are the day of the month, the day of the week and the week number (in numerical terms), respectively.
verboseLevel - the mode of "talkative", gives information about where the archive will be recorded, and so on. It is useful when you make major changes to the structure of the script.
tmpDir is the location to save the temporary file. By default, 7-Zip builds a new base file of the archive in the same directory as the old base file of the archive. By defining this key, you can set the working directory where the temporary archive base file will be built. After the temporary archive base file is built, it is copied over the original one; then the temporary file is deleted.
Differential backup
By default, files that are not in the base archive, as well as newer files are placed in the differential backup. If desired, this behavior can be changed. Key descriptions are in the standard 7z help file.
This approach has its advantages (which I have already managed to evaluate) - the speed of deployment from the backup. In the case of a full recovery, expand the last (or any other) existing differential backup, and then the full archive, refusing to overwrite existing files.
Some command line options
-bsp2 - displays a string with the progress of the execution in STDERR. STDOUT 7z is redirected to the log, progress, of course, is not written there. This command displays it in STDERR, for more information.
-ssw - packs files opened for writing by another application. If this key is not installed, 7-Zip does not include such files in the archive.
-slp is an extremely useful option. Large page mode increases compression speed. However, there is a pause at the beginning of the compression, while 7-Zip distributes large pages in memory. If 7-Zip cannot accommodate large pages, it places the usual small pages. In addition, the Task Manager does not show the actual use of memory by programs if 7-Zip uses large pages. This feature only works on Windows 2003 / XP x64. You also need to have administrator rights for your system. The recommended RAM size for this feature is 3 GB or more. if you use the
-slp mode, your Windows system may hang for a few seconds when 7-Zip allocates blocks of memory. When Windows tries to allocate large pages from RAM for 7-Zip, Windows may suspend other tasks for this time. This may look like a complete system hang, but then its operation is restored, and if the distribution is successful, 7-Zip works faster. Do not use the
-slp mode if you do not want other tasks to be “hung”. In addition, it
makes no sense to use the
-slp mode to compress small data sets (less than 100 MB). But if you compress large data sets (300 MB or more) with the LZMA method with a large dictionary, you can get an increase in speed of 5% -10% in the
-slp mode.
-mmt = on - sets multithreading mode. If you have a multiprocessor / multicore system, you can get an increase in speed with this key. 7-Zip supports multithreading mode only for LZMA / LZMA2 compression and BZip2 compression / decompression.
-ms = off - disables the creation of solid-archives. The compression quality, of course, falls, but there are very significant advantages - you can periodically update the data of the basic archive to reduce the size of the differential backups, and since the archive is not complete, you will not need to “pinch” it further. The non-solid archive is more resistant to damage, and the extraction time is much faster.
Include / exclude sheets
By default, 2 types of list are defined - the list of included files / directories (
include_general.txt ), and 2 exclusion lists (
exclude_general.txt ,
exclude_regexp.txt ).
The include list also supports UNC paths. In order to place a file / directory in the exceptions, the path must be relative.
For example, if the directory for the backup is E: \ foo \ bar, and we want to exclude the subdirectory E: \ foo \ bar \ somefolder, then in exclude_general.txt we should add bar \ somefolder or bar \ somefolder \
A path without a slash at the end can refer to both the file and the directory.
In exclude_regexp.txt are excluded to regexp files that are viewed recursively. * - a sequence of arbitrary characters? - any character.
7-Zip does not use the system wildcard parser, so "any file" for 7 Zip is '*', and '*. *' Is a file with an extension.
Finally, the whole script:
@ ECHO OFF REM Sources were found on http://sysadminwiki.ru/wiki/___Windows CD %~dp0 TITLE winfsbackup MODE CON: COLS=120 LINES=55 ECHO Setting vars... REM --- Definition block --- SET verboseLevel=1 SET tmpDir=D:\winfsbackup\tmp SET run_7z=D:\winfsbackup\7z\7za.exe SET run_pdate=D:\winfsbackup\pdate\pdate.exe FOR /F "usebackq" %%a IN (`%run_pdate% e`) DO (SET dm=%%a) FOR /F "usebackq" %%a IN (`%run_pdate% u`) DO (SET dw=%%a) FOR /F "usebackq" %%a IN (`%run_pdate% V`) DO (SET wn=%%a) SET LogDir=D:\winfsbackup\log SET Log=%LogDir%\general.log SET dDir=D:\winfsbackup\backup SET dlmDir=D:\winfsbackup\backup\old SET baseArch=%dDir%\general.7z SET IncludeList=lists\include_general.txt SET ExcludeList=lists\exclude_general.txt SET ExcludeRegexp=lists\exclude_regexp.txt SET updArch_dw=%dDir%\day_general_%dw%.7z SET updArch_wn=%dDir%\week_general_%wn%.7z IF %verboseLevel%==0 GOTO Main ECHO Verbose mode ON! ECHO Today is %wn% week of year, %dw% day of week. ECHO Full quarter backup will execute (if enabled) on 2, 19 and 36 week, friday. ECHO Temporary directory is %tmpDir% ECHO Now logging into %Log% ECHO Current backup directory is %dDir%, older backups stored into %dlmDir% :Main REM Here discribed conditions - in which case script will make new backup, update older one, etc REM You are free to change these conditions REM Make sure you envisaged all possible cases REM Actions here are not disigned to be active more than 1 at same time, excluding base condition REM If you want multiple conditions, you should edit it REM Base condition - full backup will be created if it is not exist IF NOT EXIST %baseArch% GOTO BaseArchive REM Command below turns on making full backup at 1'st day of every month, in other days - increments REM IF %dm% EQU 1 GOTO BaseArchive ELSE GOTO UpdateArchive REM This option enables updating full backup every month IF %dm% EQU 1 GOTO UpdateBase ELSE GOTO UpdateArchive REM Uncomment these 3 commands if you want to run full backup ~every quarter (2, 19, 36 week of year) REM IF NOT %wn%.%dw% EQU 02.5 GOTO UpdateArchive REM IF NOT %wn%.%dw% EQU 19.5 GOTO UpdateArchive REM IF NOT %wn%.%dw% EQU 36.5 GOTO UpdateArchive REM This option enables rewriting base archive every saturday with new files in order to decrease size of increments REM IF %dw% EQU 6 (GOTO UpdateBase) ELSE (GOTO UpdateArchive) REM Here you can place default action if conditions of previous ones were not executed. %run_pdate% "Z --- \A\c\t\i\o\n \w\a\s\ \n\o\t \s\e\l\e\c\t\e\d\! >> %Log% ECHO Warning! No one condition matching, check :Main block of script GOTO End :BaseArchive ECHO Clear %dlmDir% and move data of previous month to that dir... IF NOT EXIST %dlmDir%\nul MKDIR %dlmDir% DEL /Q %dlmDir%\* MOVE /Y %dDir%\* %dlmDir% 2> nul %run_pdate% "====== YB =======" > %Log% %run_pdate% "Z --- \S\t\a\r\t \t\o \c\r\e\a\t\e \n\e\w \a\r\c\h\i\v\e" >> %Log% ECHO Creating new backup %baseArch% %run_7z% a %baseArch% -w%tmpDir% -i@%IncludeList% -x@%ExcludeList% -xr@%ExcludeRegexp% -bsp2 -ssw -slp -scsWIN -mmt=on -mx3 -ms=off >> %Log% IF %ERRORLEVEL%==0 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \0 \- \a\r\c\h\i\v\e \s\u\c\c\e\s\s\f\u\l\l\y \c\r\e\a\t\e\d!" >> %Log% ) ELSE ( IF %ERRORLEVEL%==1 ( %run_pdate% "Z --- \W\a\r\n\i\n\g\! \R\e\c\i\e\v\e\d\ \e\x\i\t \c\o\d\e \1" >> %Log% ) ELSE ( IF %ERRORLEVEL%==2 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \2 \- \F\A\T\A\L \E\R\R\O\R\!" >> %Log% ) ELSE ( IF %ERRORLEVEL%==7 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \7 \- \C\o\m\m\a\n\d \p\r\o\m\p\t \e\r\r\o\r!" >> %Log% ) ELSE ( IF %ERRORLEVEL%==8 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \8 \- \N\o\t \e\n\o\u\g\h \m\e\m\o\r\y" >> %Log% ) ELSE ( ECHO Recieved error 255 - user stopped running process or exit code unknown! >> %Log% ) ) ) ) ) ) ) GOTO End :UpdateBase ECHO Refreshing base archive ECHO ******* ******* ******* >> %Log% %run_pdate% "Z --- \S\t\a\r\t \t\o \u\p\d\a\t\e \a\r\c\h\i\v\e" >> %Log% %run_7z% u %baseArch% -up0q1r2x1y2z1w0 -w%tmpDir% -i@%IncludeList% -x@%ExcludeList% -xr@%ExcludeRegexp% -bsp2 -ssw -slp -scsWIN -mmt=on -mx5 -ms=off >> %Log% IF %ERRORLEVEL%==0 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \0 \- \u\p\d\a\t\e \s\u\c\c\e\s\s\f\u\l\l\y \f\i\n\i\s\h\e\d" >> %Log% ) ELSE ( IF %ERRORLEVEL%==1 ( %run_pdate% "Z --- \W\a\r\n\i\n\g\! \R\e\c\i\e\v\e\d\ \e\x\i\t \c\o\d\e \1" >> %Log% ) ELSE ( IF %ERRORLEVEL%==2 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \2 \- \F\A\T\A\L \E\R\R\O\R\!" >> %Log% ) ELSE ( IF %ERRORLEVEL%==7 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \7 \- \C\o\m\m\a\n\d \p\r\o\m\p\t \e\r\r\o\r!" >> %Log% ) ELSE ( IF %ERRORLEVEL%==8 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \8 \- \N\o\t \e\n\o\u\g\h \m\e\m\o\r\y" >> %Log% ) ELSE ( ECHO Recieved error 255 - user stopped running process or exit code unknown! >> %Log% ) ) ) ) ) ) ) GOTO End :UpdateArchive ECHO Updtaing existing full backup ECHO ******* ******* ******* >> %Log% %run_pdate% "Z --- \S\t\a\r\t \t\o \u\p\d\a\t\e \a\r\c\h\i\v\e" >> %Log% IF %dw%==7 (SET updArch=%updArch_wn%) ELSE SET updArch=%updArch_dw% REM --- Check files existence --- IF EXIST %updArch% DEL /Q %updArch% REM --- Create incremental archive --- <nul set /p strTemp=Updating %baseArch% to incremental %updArch% archive ECHO. %run_7z% u %baseArch% -u- -up0q0r2x0y2z0w0!%updArch% -w%tmpDir% -i@%IncludeList% -x@%ExcludeList% -xr@%ExcludeRegexp% -bsp2 -ssw -slp -scsWIN -mmt=on -mx5 -ms=off >> %Log% IF %ERRORLEVEL%==0 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \0 \- \u\p\d\a\t\e \s\u\c\c\e\s\s\f\u\l\l\y \f\i\n\i\s\h\e\d" >> %Log% ) ELSE ( IF %ERRORLEVEL%==1 ( %run_pdate% "Z --- \W\a\r\n\i\n\g\! \R\e\c\i\e\v\e\d\ \e\x\i\t \c\o\d\e \1" >> %Log% ) ELSE ( IF %ERRORLEVEL%==2 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \2 \- \F\A\T\A\L \E\R\R\O\R\!" >> %Log% ) ELSE ( IF %ERRORLEVEL%==7 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \7 \- \C\o\m\m\a\n\d \p\r\o\m\p\t \e\r\r\o\r!" >> %Log% ) ELSE ( IF %ERRORLEVEL%==8 ( %run_pdate% "Z --- \E\x\i\t \c\o\d\e \8 \- \N\o\t \e\n\o\u\g\h \m\e\m\o\r\y" >> %Log% ) ELSE ( ECHO Recieved error 255 - user stopped running process or exit code unknown! >> %Log% ) ) ) ) ) ) ) :End ECHO Done! %run_pdate% "Z --- \D\o\n\e" >> %Log% ping localhost -w 1000 -n 5 > nul
Instead of ending
There are always people who are against such decisions, and with all their hands and feet “for” the use of all sorts of acronis and other “terry enterprises”.
I think that the solution should be commensurate with the task, and in my case the task is to have a backup file file in a secluded place that can be quickly deployed - this is what disappointed ntbackup.
The collected example can be viewed on
YandexDisk .
There is also a zip-archive for download.
Constructive criticism, advice, and even more so, testing - welcome!
Thanks for attention! All long uptime, stable link, and of course, backups at hand.