Organization of backups on different accounts of J.Disk
Greetings dear Habroskoobschestvo! The theme of backups of sites on cloud storage is quite widely disclosed. But to the question: is it worth it to reinvent the 1001st bicycle, I always answer - it is worth it.
Typical for many admins situation:
And why not organize backups on Ya.Diski our customers? Backups not only with files, but also with database dumps? And to instruct them:God forbid us, KAMAZ will leave for the oncoming lane , your websites are always on your Y.Disk.
')
Since we are talking about Yandex, we of course delegated our domains, on our server, to Yandex. Why load your server? Let Yandex work with mail, using the smtp protocol. And at the same time let Yandex take over the management of NS. Convenient, in fact.
What we need to organize backups for different accounts of Ya.Disk:
And of course we have the cherished mail of the type root@your-site.ru created in Yandex. On behalf of root@your-site.ru, we will receive letters about backups.
The algorithm proposed in the article was tested on a server with Ubuntu 16.04 on board.
If you already send yourself an email from the server, then skip this item, otherwise read.
Since we have a lot of users on the server, we’ll have to work as root superuser. Moreover, you will have to register the database users and database passwords, and let all this data be stored in the / root directory.
We closed the ssh-connection for the root? We have only one, the most trusted user can perform su?
Perform in the console:
We moved to the / root directory and installed ssmtp. All further commands will also be executed in this directory.
We edit the ssmtp config:
We bring it to the form:
The config is obvious, there is nothing to comment on.
Everything! This is enough to solve our problem.
Let's create in / root the template of the letter about our backups, and check the work of ssmtp
We bring it to mind:
We send a letter:
There should be no problems, the setup is elementary.
I use for work the client Ya.Disk from the developer Anton Batenev: github.com/abbat/ydcmd
Why was this client chosen?
To install ydcmd, run the commands:
The client we are looking at is nothing more than a python script - ydcmd.py, copied to / usr / bin. To work, we need to get tokens for Yandex accounts of our clients.
It makes no sense to describe the process of obtaining tokens. Details on Gitkhab: github.com/abbat/ydcmd , or in the Yandex manual .
After receiving the tokens, we will create different configs for our clients. I have two different tokens, one common to all sites, the second for an individual client, with whom a service contract has been concluded. Therefore, the article discusses two configs: for all users, and for the user vet.
Create a default config in / root
We bring it to the form:
By analogy, we create a config for the user vet, the sites of this user will be backed up to another Ya.Disk.
We also bring it to the form:
Let's check the work of ydcmd for both configs:
In both cases, something like this should be displayed:
How it usually works:
Create folders for backups on Ya.Diski:
On this with the preparatory part - everything.
Works ssmtp, created a letter template about backups, configured client Ya.Disk for different accounts. I remind you that we are working in the / root folder.
Actually there is nothing new in the bash script. Compilation from various sources that I do not remember. The script works every other day, the last three copies are saved. Why is that? Because it's more convenient for me. Everyone can modify it for themselves.
Actually it remains to register your variables here:
And here in the case:
Only it is not clear what the variable FILE
The variable FILE is nothing more than a csv file, like:
For each backup we create our own csv file with your own data:
system_user_name; domain name; DB_for_dampa; db_user_name; password_db
Naturally, the name of the system user coincides with the name of the home directory of this user. User vet - home directory is also vet.
Let's create the necessary csv-files for our backups, tracking the absence of extra empty lines in the file ! There are no checks for empty lines in the script!
Those. When creating a csv file, the cursor, when saved, should be at the end of the last line .
Everything is ready for a trial run.
Check everything again:
We work in the root folder. We try to run a script for a user with a small amount of data, I have this test user vet, as a parameter we pass the user name to the script:
Since everything is transparent and simple, there should be no problems.
For error:
We do not pay attention.
But there may be an ambush: the script can simply stop working and do nothing.
Look at the lines:
And we change 1 to 0, and again we try to start.
What is going to happen:
Actually everything. It remains to register tasks in cron.
It looks like this to me:
As we can see, cron works at one in the morning and at three in the morning for different backups. The script itself will work every other day.
That's all, waiting for constructive criticism, comments and clarifications.
Typical for many admins situation:
- there is a virtual / non-virtual server;
- There are several clients that you support since you created the site;
- There is a problem backups of your clients sites.
And why not organize backups on Ya.Diski our customers? Backups not only with files, but also with database dumps? And to instruct them:
')
Since we are talking about Yandex, we of course delegated our domains, on our server, to Yandex. Why load your server? Let Yandex work with mail, using the smtp protocol. And at the same time let Yandex take over the management of NS. Convenient, in fact.
What we need to organize backups for different accounts of Ya.Disk:
- install and configure ssmtp on your server;
- install and configure the necessary client Ya.Disk;
- create and configure a bash script for backup;
- register tasks in cron.
And of course we have the cherished mail of the type root@your-site.ru created in Yandex. On behalf of root@your-site.ru, we will receive letters about backups.
The algorithm proposed in the article was tested on a server with Ubuntu 16.04 on board.
Install, configure, test ssmtp on our server
If you already send yourself an email from the server, then skip this item, otherwise read.
Since we have a lot of users on the server, we’ll have to work as root superuser. Moreover, you will have to register the database users and database passwords, and let all this data be stored in the / root directory.
We closed the ssh-connection for the root? We have only one, the most trusted user can perform su?
Perform in the console:
su cd /root apt update && apt upgrade apt install ssmtp We moved to the / root directory and installed ssmtp. All further commands will also be executed in this directory.
We edit the ssmtp config:
nano /etc/ssmtp/ssmtp.conf We bring it to the form:
root=root@your_site.ru mailhub=smtp.yandex.ru:465 AuthUser=root@your_site.ru AuthPass=your_passwd AuthMethod=LOGIN FromLineOverride=YES UseTLS=YES Hostname=your_site.ru The config is obvious, there is nothing to comment on.
Everything! This is enough to solve our problem.
Let's create in / root the template of the letter about our backups, and check the work of ssmtp
nano rsync_email_orig We bring it to mind:
To: your_email@gmail.com From: root@your-site Subject: Backup sites Info on backup: We send a letter:
cp rsync_email_orig rsync_email_orig_test echo 'Test msg' >> rsync_email_orig_test ssmtp your_email@gmail.com < rsync_email_orig_test rm -f rsync_email_orig_test There should be no problems, the setup is elementary.
Install, customize client Ya.Disk
I use for work the client Ya.Disk from the developer Anton Batenev: github.com/abbat/ydcmd
Why was this client chosen?
- you can work with different Yandex accounts;
- you can synchronize only what you need.
To install ydcmd, run the commands:
sudo add-apt-repository ppa:abbat/ydcmd sudo apt update sudo apt install ydcmd The client we are looking at is nothing more than a python script - ydcmd.py, copied to / usr / bin. To work, we need to get tokens for Yandex accounts of our clients.
It makes no sense to describe the process of obtaining tokens. Details on Gitkhab: github.com/abbat/ydcmd , or in the Yandex manual .
After receiving the tokens, we will create different configs for our clients. I have two different tokens, one common to all sites, the second for an individual client, with whom a service contract has been concluded. Therefore, the article discusses two configs: for all users, and for the user vet.
Create a default config in / root
nano /root/.ydcmd.cfg We bring it to the form:
[ydcmd] token = 1234567890 By analogy, we create a config for the user vet, the sites of this user will be backed up to another Ya.Disk.
nano /root/.ydcmd_vet.cfg We also bring it to the form:
[ydcmd] token = any_token Let's check the work of ydcmd for both configs:
ydcmd stat ydcmd --config=.ydcmd_vet.cfg stat In both cases, something like this should be displayed:
name: disk exif: {} resource_id: 21239186:e9065863c345ergdfghjfgy51da3c5e06bc12345afeb14158ddcaae created: 2012-04-04T20:00:00+00:00 modified: 2012-04-04T20:00:00+00:00 path: disk:/ comment_ids: {} type: dir revision: 1354646733351472 How it usually works:
ydcmd --help Create folders for backups on Ya.Diski:
ydcmd mkdir backup ydcmd --config=.ydcmd_vet.cfg mkdir backup On this with the preparatory part - everything.
Works ssmtp, created a letter template about backups, configured client Ya.Disk for different accounts. I remind you that we are working in the / root folder.
Preparing bash script for backups
Actually there is nothing new in the bash script. Compilation from various sources that I do not remember. The script works every other day, the last three copies are saved. Why is that? Because it's more convenient for me. Everyone can modify it for themselves.
bash script
#!/bin/bash DATE=`/bin/date '+%d.%m.%Y'` USER=$1 DIR=/var/www DIR_SITES=public_html EMAIL=your_email@gmail.com TEMPLATE_EMAIL=/root/rsync_email D=`date +%j` # . 2 bash. == 1, == 0. .. if [ $((10#$D % 2)) == 1 ];then case $USER in vet) FILE=user_list_vet cnf=.ydcmd_vet.cfg DIR_BACKUP=/root/backup_vet ;; *) FILE=user_list_all cnf=.ydcmd.cfg DIR_BACKUP=/root/backup ;; esac else exit 1 fi if ! [[ -d $DIR_BACKUP ]]; then mkdir $DIR_BACKUP fi cp rsync_email_orig rsync_email echo "$DATE" >> $TEMPLATE_EMAIL echo "==============================================" >> $TEMPLATE_EMAIL while read line;do IFS=";" set -- $line USER=$1 SITES=$2 DB=$3 DB_USER=$4 DB_PASSWD=$5 FILE_NAME=$DIR_BACKUP/backup_"$SITES"_"$DATE".zip cd $DIR/$USER/$DIR_SITES mysqldump -u$DB_USER -p$DB_PASSWD $DB > $DB.sql zip -r9 $FILE_NAME $SITES $DB.sql > /dev/null rm ./$DB.sql zip -T $FILE_NAME >> $TEMPLATE_EMAIL echo "==============================================" >> $TEMPLATE_EMAIL chown $USER:$USER $FILE_NAME done < /root/$FILE find $DIR_BACKUP/ -mtime +4 -exec rm -f {} \; /usr/bin/ydcmd --config=/root/$cnf put --rsync $DIR_BACKUP/ disk:/backup >> $TEMPLATE_EMAIL /usr/sbin/ssmtp $EMAIL < $TEMPLATE_EMAIL exit 0 Actually it remains to register your variables here:
#home dir DIR=/var/www #sites dir DIR_SITES=public_html EMAIL=your_email@gmail.com And here in the case:
if [ $((10#$D % 2)) == 1 ];then case $USER in vet) FILE=user_list_vet cnf=.ydcmd_vet.cfg DIR_BACKUP=/root/backup_vet ;; *) FILE=user_list_all cnf=.ydcmd.cfg DIR_BACKUP=/root/backup ;; esac else exit 1 fi Only it is not clear what the variable FILE
FILE=user_list_vet The variable FILE is nothing more than a csv file, like:
user1;site1.ru;DB_name1;user_DB_1;passwd_DB_1 user1;site2.ru;DB_name2;user_DB_2;passwd_DB_2 user2;site3.ru;DB_name3;user_DB_3;passwd_DB_3 ... For each backup we create our own csv file with your own data:
system_user_name; domain name; DB_for_dampa; db_user_name; password_db
Naturally, the name of the system user coincides with the name of the home directory of this user. User vet - home directory is also vet.
Let's create the necessary csv-files for our backups, tracking the absence of extra empty lines in the file ! There are no checks for empty lines in the script!
Those. When creating a csv file, the cursor, when saved, should be at the end of the last line .
Everything is ready for a trial run.
Check everything again:
- there is a letter template rsync_email_orig and ssmtp works
- Ya.Disk's ydcmd client is configured for different configs
- edited all variables in rsync.sh
- created different csv files for different backups
We work in the root folder. We try to run a script for a user with a small amount of data, I have this test user vet, as a parameter we pass the user name to the script:
./rsync.sh vet Since everything is transparent and simple, there should be no problems.
For error:
mysqldump: [Warning] Using a password on the command line interface can be insecure. We do not pay attention.
But there may be an ambush: the script can simply stop working and do nothing.
Look at the lines:
# . 2 bash. == 1, == 0. .. if [ $((10#$D % 2)) == 1 ];then And we change 1 to 0, and again we try to start.
What is going to happen:
- in the folder specified in the script for backups, an archive should be created with a backup of the form: backup_your_site.ru_date.zip
- backup folder will be synchronized with Ya.Disk for the desired account
- we will receive a letter of the form:
Info on backup: Fri May 12 07:49:15 MSK 2017 ============================================== test of /root/backup_vet/your-site_date.zip OK ==============================================
Actually everything. It remains to register tasks in cron.
It looks like this to me:
su crontab -e 00 01 * * * /bin/bash /root/rsync.sh all 00 03 * * * /bin/bash /root/rsyn.sh vet As we can see, cron works at one in the morning and at three in the morning for different backups. The script itself will work every other day.
That's all, waiting for constructive criticism, comments and clarifications.
Source: https://habr.com/ru/post/328578/
All Articles