Fast backup implementation in Amazon S3

I am tired of experiencing certain concerns about the safety of data on a dedicated server, watching what is happening with the hosts recently ... 3FN, Agava, Hosting.UA, McHhost - the trend is very bad.

As a result, McHost became the last straw, and, in order not to be among those who are “already making backups”, I transferred the backup system of my server to Amazon S3. It turned out quite a quick and transparent.

I want to share with the public the simplest implementation.

1. First of all, remembering that the service is paid, we calculate the costs
   Let's use the calculator , kindly provided to us by Amazon.
   In the menu on the left, select Amazon S3 , remove the tick from FREE INBOUND and fill in the required fields (everything is extremely simple). Having filled, we press Add to bill and we study the price in a window on the right. So, for example, when storing 100 Gb of data, uploading 20 Gb per month and downloading 2 Gb per month we pay about $ 17, which for me is quite an acceptable price for peace of mind. If the volumes you need are an order of magnitude smaller (which, I suspect, it will be so), then the fee becomes equal to several dollars. I would venture to suggest that it will not be too stressful for the budget. If you think the same way, go on.
   
2. Subscribing to Amazon Simple Storage Services (S3)
   On the service page, click Sign Up For Amazon S3 .
   If you do not have an account in Amazon Web Services, select I am a new user , otherwise enter your account password in the proposed form.
   To register with AWS, enter your name, email address, password and create an account.
   The next page will be the registration itself in S3: re-enter the mailing address, accept the agreement and get to the page with information about tariffs and the choice of payment method.
   We drive in the data of our credit card and billing address. Once again we look around the completed questionnaire and click Complete Sign Up .
   Now it remains to wait a bit until an email arrives in the mail confirming that the account is activated and available.
   
3. We get the access keys to the service
   We go to the account security section and in the Access Credentials -> Access Keys section we see the key we need for authorization (the Access Key ID pair is the Secret Access Key). We need it when setting up backup on the server.
   
4. Install the utility to work with S3
   For Debian and Ubuntu are in the repositories: apt-get install s3cmd
   
5. Perform initial utility configuration
   Under that user, from under which we will perform further backup:
   s3cmd --configure
   Enter the Access Key, Secret Key, password for encryption, the path to the GPG program.
   Choose to use or not HTTPS and proxy.
   After entering all the settings, the configurator will offer to test the connection. We agree, and after successfully passing the test, we agree to save the settings in the configuration file.
   Now s3cmd is ready to go.
   
6. Prepare S3
   Create a repository, the so-called bucket:
   # s3cmd --acl-private --bucket-location=EU mb s3://mybucketname
Bucket 'mybucketname' created
   --acl-private - access only for us
   --bucket-location = EU - the bucket is created in the European repository (from Moscow, where I have a server, will be closer). If the US is closer to you, then instead of the EU we write US.
   mybucketname is the name of the repository, it must be unique among all (not only your) repositories.
   ')
   Checking:
   # s3cmd la
Bucket 'mybucketname':
   
7. Actually, backup
   s3cmd has an rsync mode of operation, which in the case of the simplest backup is very convenient:
   s3cmd --acl-private --bucket-location=EU --guess-mime-type --delete-removed sync /local/backup/dir/ s3://mybucketname/backupfolder
   What is what:
   The first two parameters are described above.
   --guess-mime-type — select the content MIME type based on the widening.
   - delete -removed - delete files in the storage if they were deleted from the local folder.
   Next is the local folder in which the backups on the server lie (you are doing local backups, right?) And the folder into which to backup to s3 (if it isn’t already there, it will be created automatically).
   
   We substitute the parameters you need, we launch, we check that everything is being downloaded to the right place:
   s3cmd ls s3://mybucketname
   
   UPD: File size in S3 should not exceed 5 Gb. If your backup system makes files larger, then it will have to be slightly modified, for example, skipping archives through split. Thank you nightfog for covering this issue.
   
   
8. We add the launch of the synchronization command to your system for manufacturing local backups or simply to cron (as I did).
   
9. PROFIT!
   

How to recover data?
To restore all data from S3, in the synchronization team, we swap the source and target, not forgetting to remove the missing files:
s3cmd sync s3://mybucketname/backupfolder /local/backup/dir/
There is a synchronization in the opposite direction and voila, all data is again on the local disk.

Or, to restore a single file, use the get command:
s3cmd get s3://mybucketname/backupfolder/backup.tar.gz /local/backup/dir/

UPD2: Thanks for the karma, transferred to the system administration.