A simple version of the backup system in Python, Bash and Git
Recently, there is some discomfort when I start work. The feeling was not so strong, but the concentration hindered. I thought laziness. It turned out that everything is a little more difficult :) The laptop I work for is almost 3 years old; Mac OS X 10.6.1 is on it, but there are no apples on it anywhere, and there is no periodic backup system on it as a class. In general, there was no sense of stability and reliability, so I took up this issue closely. Actually, then I will describe the result that my subconscious mind satisfied :) Maybe something will be useful to someone.
Of course, I sometimes have photos that I would like to keep, and other binaries, but we will not touch on the issue of backup, because I don’t have a safe place to stuff more than 10 GB with scripts, and more often than once per month they do not change normally.
Perhaps in the near future, this list will expand, but for now, as it seems to me, this is more or less sufficient.
')
By my requests, Google did not give anything worthwhile and satisfying to my requests, so I had to write it myself. At the same time I touched Python, I’ve been looking for a suitable occasion for a long time.
The option "copy all" does not work due to lack of space. In addition, it somehow seems silly to keep a bunch of backup copies of unnecessary files (object logs, build logs, etc.). In the end, so to speak, the terms of reference were as follows:
The copy script is written in Python and takes slightly less than 200 lines (including indents, comments, etc.). Additionally, it uses one fuzzy module to convert absolute paths into relative paths and vice versa.
Using:
Limitations:
Example list (with base directory ~):
Source codes (probably, it is more convenient when they lie on file hosting than when those same 200 lines litter the text):
sn-backup.py ( look at dumpz.org )
relpath.py ( view on dumpz.org )
relpath.py was taken from here , from the first comment, and slightly corrected.
Compression and encryption require one line at a time, so it would be somehow strange to write them in Python. How to compress - a matter of taste. I prefer 7-zip. The command line here, accordingly, will be simple (the archive name is backup.7z, the folder name is .backup):
I chose openssl for encryption, since, in theory, almost everywhere is and works out of the box, without the need to generate anything. The command looks like this:
Accordingly, the name of the source file backup.7z, the name of the output file backup.aes256cbc (in case I suddenly forget what the encryption algorithm is called). The password is stored in a file and can be of any length. Better, though, at least 32 characters.
Due to the potential variety of methods, I first wanted to send it through a highly customizable Python script, but then I decided that copying to a small number of hosts is much easier manually, and copying to a large number of hosts smacks of paranoia. Therefore, sending is implemented in the sh-script. The Copybox command on Dropbox, when the client is installed, is of no interest, so consider only the command to send to the mail (it is even less common):
Attention: for this command to work, you need to configure an SMTP server on your local machine. My provider provides it. The team attaches an encrypted archive to a message with the subject “[BACKUP] <current date and time>” and sends a message to <address>. I have this command in the sn-upload.sh script, which takes the name of the encrypted archive file as the only argument.
Everything is going together by the backup.sh script, which looks like this:
At leisure, perhaps, I will make such improvements:
I do not exclude the possibility that I invented the bicycle. But I didn’t spend much time on it, and got a little experience. I want to note that above all in, ahem, the design of this system laid simplicity. I know what files and from where I need to copy. I know what to do with them then. I need a script only to make it all on the machine. In my opinion, he performs his tasks completely.
Thank you for attention!
Task
What we backup
- Catalogs with various sources. Some projects are Git-repositories, respectively, I want to save the repository, and not just the latest version of the source code. NB: I was somehow not used to run make clean or analogs at the end of the day, so we add the requirement that binaries are not copied.
- Catalogs with configs and parameters.
- Directories with TeX sources. In fact, one could add to the first paragraph.
Of course, I sometimes have photos that I would like to keep, and other binaries, but we will not touch on the issue of backup, because I don’t have a safe place to stuff more than 10 GB with scripts, and more often than once per month they do not change normally.
Where to backup
- Actually, file storage. A free Dropbox account was chosen, because copying with the cp command is easy and convenient.
- Mail to Gmail. All the same, 7 gigs are not used, and there has been no reason to doubt the reliability yet.
Perhaps in the near future, this list will expand, but for now, as it seems to me, this is more or less sufficient.
')
What are the features
- Simplicity. The main feature :) No need for complex and large programs, I myself am able to decide which files from where I need to copy. It is easier to be confident in the correctness of the work of a small program.
- History of changes. The first feature you want is a change history, so as not to swear on the topic “what this file prevented me from!”
- Compression. Git, of course, presses, but because it is still more convenient to store backups in one file, so let someone else do it.
- Encryption I do not doubt the honesty and sekyurnosti Internet providers and storage providers. However, I don’t want that what I do is transmitted and it is not clear where in open form.
Decision
By my requests, Google did not give anything worthwhile and satisfying to my requests, so I had to write it myself. At the same time I touched Python, I’ve been looking for a suitable occasion for a long time.
Copying
The option "copy all" does not work due to lack of space. In addition, it somehow seems silly to keep a bunch of backup copies of unnecessary files (object logs, build logs, etc.). In the end, so to speak, the terms of reference were as follows:
- The list of directories and files to copy is stored in a separate file.
- For each directory, you can specify the file masks that you want to copy from it. If the mask is not specified, copy all.
- The original directory structure is maintained.
- The script copies (and only copies) files into the specified folder.
- If the file already exists in the target folder, copy only if the modification time of the source file is longer.
The copy script is written in Python and takes slightly less than 200 lines (including indents, comments, etc.). Additionally, it uses one fuzzy module to convert absolute paths into relative paths and vice versa.
Using:
sn-backup.py <file with list> <base directory> <backup directory>
Limitations:
- Strict file format with a list. Line format:
<path relative to base directory> [\ t <masks separated by commas>]
More than one tab is not allowed. Masks are separated by commas without spaces. - Virtually no fool protection.
Example list (with base directory ~):
.emacs Documents / Programming * .c, *. H, *. Cpp, *. Pro, *. Py, .git Scripts
Source codes (probably, it is more convenient when they lie on file hosting than when those same 200 lines litter the text):
sn-backup.py ( look at dumpz.org )
relpath.py ( view on dumpz.org )
relpath.py was taken from here , from the first comment, and slightly corrected.
Compression and encryption
Compression and encryption require one line at a time, so it would be somehow strange to write them in Python. How to compress - a matter of taste. I prefer 7-zip. The command line here, accordingly, will be simple (the archive name is backup.7z, the folder name is .backup):
7za a -mx7 backup.7z .backup
I chose openssl for encryption, since, in theory, almost everywhere is and works out of the box, without the need to generate anything. The command looks like this:
openssl enc -aes-256-cbc -salt -in backup.7z -out backup.aes256cbc -pass file: <pass path>
Accordingly, the name of the source file backup.7z, the name of the output file backup.aes256cbc (in case I suddenly forget what the encryption algorithm is called). The password is stored in a file and can be of any length. Better, though, at least 32 characters.
Sending
Due to the potential variety of methods, I first wanted to send it through a highly customizable Python script, but then I decided that copying to a small number of hosts is much easier manually, and copying to a large number of hosts smacks of paranoia. Therefore, sending is implemented in the sh-script. The Copybox command on Dropbox, when the client is installed, is of no interest, so consider only the command to send to the mail (it is even less common):
uuencode <path to encrypted archive> $ (basename <path to encrypted archive>) | mail -s "[BACKUP] $ (date)" <address>
Attention: for this command to work, you need to configure an SMTP server on your local machine. My provider provides it. The team attaches an encrypted archive to a message with the subject “[BACKUP] <current date and time>” and sends a message to <address>. I have this command in the sn-upload.sh script, which takes the name of the encrypted archive file as the only argument.
Together
Everything is going together by the backup.sh script, which looks like this:
# ~/.backup - , .
# .git-.
cd ~/.backup
# .
~/Scripts/sn-backup.py ./list ~ .
# , git.
git add .
git commit -m "Backup at $(date)"
# .
cd ..
7za a -mx7 backup.7z .backup > /dev/null
# .
openssl enc -aes-256-cbc -salt -in backup.7z -out backup.aes256cbc -pass file:///${SECRET_PATH}/.password
# .
~/Scripts/sn-upload.sh ./backup.aes256cbc
# ???
rm backup.7z
# PROFIT!
Directions for further development
At leisure, perhaps, I will make such improvements:
- Translate all bash scripts to Python.
- It is possible to switch to pylzma and Python built-in cryptography (for cross-platform).
- Make the file format list softer and more decent.
Conclusion
I do not exclude the possibility that I invented the bicycle. But I didn’t spend much time on it, and got a little experience. I want to note that above all in, ahem, the design of this system laid simplicity. I know what files and from where I need to copy. I know what to do with them then. I need a script only to make it all on the machine. In my opinion, he performs his tasks completely.
Thank you for attention!
Source: https://habr.com/ru/post/74532/
All Articles