How do we “back up” servers in Amazon and fend off piranhas
Many owners of web projects hosted on the popular cloud provider Amazon probably think about how to create an effective and reliable configuration of a web solution, how to back up machines, how to act in case of a data center collapse that hosts your servers. Unfortunately, the official documentation of cloud hosting is "somewhat" stingy on the topic of reliability and internal implementation of web services - so you have to rely on your own combat operating experience and military ingenuity.
The situation is aggravated by the currently observed confrontation of a common misconception that everything is very reliable and you can sleep soundly in the cloud: hard drives “made of titanium”, network traffic flows “through the gold wires”, and the engineers there are tough guys from Chelyabinsk dissecting the urinal in half - with cloudy recommendations provider on the topic "we provide services of sufficient reliability and fast channels between the DC, and the task of the project architect is to combine our technologies, bringing the right girls to the right amount."
In this article I want to share our observations on the behavior of services for six months, architectural solutions on EBS disks to ensure the reliability and performance of web projects and specific backup techniques that we use in our daily work.
To be honest, from the official Amazon documentation on EBS disks, I still did not understand whether they are more reliable RAID1 or not - all the talk about the functional dependence of the disk volume, the last time the disk was taken, the number of hairs in the beard of the wizard and the phases of the moon do not inspire optimism:
“It’s been your last snapshot. If you’ve seen the most recent loss rate, it’s about 0.1% - 0.5%, where it refers to the complete loss of volume.
')
During the recent accident in the European DC of Amazon, 2 drives from the software-based RAID10 immediately flew out due to a power outage in the data center. The network also sometimes shows scary messages that the EBS disk suddenly “broke down”, the client was sent a letter on the topic “your data was lost, unfortunately” and is officially recommended to snapshot just a minute before the disk fails :-) The snapshot mechanism appeared precisely to "secure" against the failure of not very reliable EBS disks.
Public virtual machines that are used to create their own private images have only one EBS disk, which is not in the raid and on which root partition. And to load a machine with a root partition on a software raid is, to put it mildly, without a file it is not easy and takes time.
However, for half a year in Amazon we have not a single disk (we have about 50), apart from the power failure after a lightning strike , which makes it possible to treat them as more or less reliable “RAID1” disks (tuk, tuk ; replication to one device located in the same data center), inferior in s3 reliability (replicates additionally to 2 devices in different data centers, and in general it is not a block device), but 10 times more reliable than regular hard drives.
After migrating to Amazon at the beginning of this year, we faced a rather low performance of EBS disks, which was especially noticeable on a machine with MySQL. Having “watched” the idea in RDS (Amazon automatically, depending on the total volume of EBS disks, put the database information on the software raid) and using military intelligence we transferred the data to the software created from EBS disks RAID10 (mdadm in Linux):
Oh yes, the spare-disk, as you can see, we use :-) 'sdj [8] (S)'. We decided to put all the same in raids to increase the level of resiliency.
After the migration of data on raids, the performance problems of the disk subsystems on the machines stopped being disturbed.
Because our main site (www.1c-bitrix.ru) works in a web cluster configuration: two web machines, two machines with a database (master + slave) - behind the balancer (nginx), we really, really wanted to use the cluster file system type OCFS2 or GFS2. But it turned out that EBS disks could not be simultaneously mounted on several virtual machines. Therefore, we use the clustering utility csync2 , which is forced by the add-in to “quickly” synchronize frequently updated heavy folders based on inotify (if you are interested, we’ll tell you more).
Virtual machine disks are easy to back up, because the tool for creating block device snepshots was created just for this:
The operation starts very quickly (seconds), and the download of the snapshot in s3 lasts for a certain time, depending on the disk size and the load on Amazon, up to tens of minutes.
We are adults and we understand that holistic disk snapshot can be if:
Otherwise, we get the disk snooping "as if after a sudden power off" and the disk will need to be treated (fsck) when the machine is booted - which, thanks to modern journaling file systems (ext3, ext4, xfs, etc. ) occurs unnoticed and often quickly due to playing the magazine. We tested the creation of a snapshot of a live disk (without unmounting) ext3 and xfs - after a pause to play the log, a disk (created from a snapshot) is mounted and running.
We chose xfs as a file system that supports “suspension”. This file system is recommended by Amazon and is used by utilities for creating a complete snapshot of the ec2-consistent-backup type :
Indeed, if after “xfs_freeze -f” you make a snapshot of a disk, it is mounted very quickly without playing the log.
In CentOS6 (and apparently in other distributions with the “updated” kernel), it became possible to “default” also default file systems like ext3 / ext4 using the fsfreeze command. We plan to test this utility thoroughly in the context of creating complete snapshots of EBS disks.
If you think logically, then you need a) “stop” the raid file system, b) start creating a snapshot FOR EACH physical raid disk, c) release the raid file system.
Unfortunately, there is no information in the Amazon API documentation about how much time must pass between b) and c). However, it was experimentally found (and similarly done in this utility ) that it is enough to get the snapshot identifiers from the API call for each raid disk and the raid file system can be “thawed” - and when it is assembled, it will pick up without rebuilding. Otherwise, it will begin to rebuild. That's what we did:
Although you can one- handedly reset MySQL data to disk: FLUSH TABLES WITH READ LOCK , freeze the file system with your left foot: xfs_freeze -f / path, and press Enter with your nose to perform a snapshot API removal call: ec2-create-snapshot ... for now arranges asynchronous replication of MySQL to another DC :-). Interesting is the creation of base snapshots using xtrabackup - without the above described dances with tambourines.
If you separately backup disks, disk groups in raids, then, if there is a config, the process works stably and transparently. However, if you sometimes need (for example, a lightning struck the DC at the weekend :-)) to move between AZ (data centers) Amazon - it is easier to enter a level of abstraction above those of instances, snapshots and disks and operate with categories of roles and images (AMI - Amazon Machine Image ) machines. Through the API, you can assign tags to Amazon objects and filter samples by them.
We defined the roles: web machine, DBMS, balancer, monitoring machine and our backup scripts work like this:
However, it is much easier to back up a running virtual machine into an image (AMI image). At the same time, its settings and paths to the created snapshots are saved in the image, from which the machine disks will be created, which are attached to it at the start:
Feel the difference! I create an image from a machine with a raid of 10 discs - one team. And I don’t bother with its discs and snapshots — all the information necessary for recreating a car is recorded into the AMI image automatically. The only drawback is that it is unclear how long it takes to drive disks / raids of a running machine in order to create an AMI image from it. Experienced to find that 2-5 seconds are enough - if less, then the disks / raids begin to recover when creating a new car. It is also very convenient that you can raise the machine from the backup to the image (AMI) in another data center (AZ).
Now we are creating a backup twice a day to the AMI image of all the machines in our cluster configuration (hundreds of gigabytes). Cleaning up "obsolete" images, along with their snapshots, is done by the script:
Amazon AWS has a nice admin area, but many things can only be done through the API. For Linux, you can install command line tools for working with Amazon APIs, which are fairly well documented. We use such tools:
The last script sets the environment variables for the operation of the tools. Please note that almost every Amazon web service has its own set of utilities that need to be downloaded and installed separately. For managing machines, for example, the documentation is here , and command line utilities are here .
So far, the official SDK for working with the Amazon API for PHP shows itself well (the above script uses this particular library).
You can work efficiently with Amazon EBS disks by combining them into a software raid. Unfortunately, you cannot mount an EBS disk to multiple virtual machines to work with a cluster file system of the type OCFS2, GFS2. Backups of both disks and EBS raids are made easy, you only need to understand the basic principles and get around the pitfalls. Conveniently backing up the machines as a whole, rather than on a separate disk - this will allow you to quickly deploy the configuration in a nearby data center if a lightning strikes or a plane crashes into your DC. Good luck to all!
The situation is aggravated by the currently observed confrontation of a common misconception that everything is very reliable and you can sleep soundly in the cloud: hard drives “made of titanium”, network traffic flows “through the gold wires”, and the engineers there are tough guys from Chelyabinsk dissecting the urinal in half - with cloudy recommendations provider on the topic "we provide services of sufficient reliability and fast channels between the DC, and the task of the project architect is to combine our technologies, bringing the right girls to the right amount."
In this article I want to share our observations on the behavior of services for six months, architectural solutions on EBS disks to ensure the reliability and performance of web projects and specific backup techniques that we use in our daily work.
Is EBS Disk Reliability - RAID1 or "Worse"?
To be honest, from the official Amazon documentation on EBS disks, I still did not understand whether they are more reliable RAID1 or not - all the talk about the functional dependence of the disk volume, the last time the disk was taken, the number of hairs in the beard of the wizard and the phases of the moon do not inspire optimism:
“It’s been your last snapshot. If you’ve seen the most recent loss rate, it’s about 0.1% - 0.5%, where it refers to the complete loss of volume.
')
During the recent accident in the European DC of Amazon, 2 drives from the software-based RAID10 immediately flew out due to a power outage in the data center. The network also sometimes shows scary messages that the EBS disk suddenly “broke down”, the client was sent a letter on the topic “your data was lost, unfortunately” and is officially recommended to snapshot just a minute before the disk fails :-) The snapshot mechanism appeared precisely to "secure" against the failure of not very reliable EBS disks.
Public virtual machines that are used to create their own private images have only one EBS disk, which is not in the raid and on which root partition. And to load a machine with a root partition on a software raid is, to put it mildly, without a file it is not easy and takes time.
However, for half a year in Amazon we have not a single disk (we have about 50), apart from the power failure after a lightning strike , which makes it possible to treat them as more or less reliable “RAID1” disks (tuk, tuk ; replication to one device located in the same data center), inferior in s3 reliability (replicates additionally to 2 devices in different data centers, and in general it is not a block device), but 10 times more reliable than regular hard drives.
Soft raid and low performance EBS-drives
After migrating to Amazon at the beginning of this year, we faced a rather low performance of EBS disks, which was especially noticeable on a machine with MySQL. Having “watched” the idea in RDS (Amazon automatically, depending on the total volume of EBS disks, put the database information on the software raid) and using military intelligence we transferred the data to the software created from EBS disks RAID10 (mdadm in Linux):
cat /proc/mdstat
Personalities : [raid10]
md0 : active raid10 sdo[0] sdj[8](S) sdi[7] sdh[6] sdg[5] sdn[4] sdm[3] sdl[2] sdk[1]
629145344 blocks 64K chunks 2 near-copies [8/8] [UUUUUUUU]
Oh yes, the spare-disk, as you can see, we use :-) 'sdj [8] (S)'. We decided to put all the same in raids to increase the level of resiliency.
After the migration of data on raids, the performance problems of the disk subsystems on the machines stopped being disturbed.
Cluster file systems and EBS disks
Because our main site (www.1c-bitrix.ru) works in a web cluster configuration: two web machines, two machines with a database (master + slave) - behind the balancer (nginx), we really, really wanted to use the cluster file system type OCFS2 or GFS2. But it turned out that EBS disks could not be simultaneously mounted on several virtual machines. Therefore, we use the clustering utility csync2 , which is forced by the add-in to “quickly” synchronize frequently updated heavy folders based on inotify (if you are interested, we’ll tell you more).
Backup EBS Discs
Virtual machine disks are easy to back up, because the tool for creating block device snepshots was created just for this:
ec2-create-snapshot vol-a5rtbvweThe operation starts very quickly (seconds), and the download of the snapshot in s3 lasts for a certain time, depending on the disk size and the load on Amazon, up to tens of minutes.
We are adults and we understand that holistic disk snapshot can be if:
- The disk is pre-unmounted: "umount -d / dev / sdh".
- The server is previously stopped.
- The file system is "frozen." Although in this case, you need to tell the applications to reset the data to disk.
Otherwise, we get the disk snooping "as if after a sudden power off" and the disk will need to be treated (fsck) when the machine is booted - which, thanks to modern journaling file systems (ext3, ext4, xfs, etc. ) occurs unnoticed and often quickly due to playing the magazine. We tested the creation of a snapshot of a live disk (without unmounting) ext3 and xfs - after a pause to play the log, a disk (created from a snapshot) is mounted and running.
"Freezing" and "freezing" of the file system
We chose xfs as a file system that supports “suspension”. This file system is recommended by Amazon and is used by utilities for creating a complete snapshot of the ec2-consistent-backup type :
xfs_freeze -f | -u mount-pointIndeed, if after “xfs_freeze -f” you make a snapshot of a disk, it is mounted very quickly without playing the log.
In CentOS6 (and apparently in other distributions with the “updated” kernel), it became possible to “default” also default file systems like ext3 / ext4 using the fsfreeze command. We plan to test this utility thoroughly in the context of creating complete snapshots of EBS disks.
Software raid backup
If you think logically, then you need a) “stop” the raid file system, b) start creating a snapshot FOR EACH physical raid disk, c) release the raid file system.
Unfortunately, there is no information in the Amazon API documentation about how much time must pass between b) and c). However, it was experimentally found (and similarly done in this utility ) that it is enough to get the snapshot identifiers from the API call for each raid disk and the raid file system can be “thawed” - and when it is assembled, it will pick up without rebuilding. Otherwise, it will begin to rebuild. That's what we did:
bxc_exec("/usr/bin/ssh remote_xfs_freezer@hostname".escapeshellarg("sudo /usr/sbin/xfs_freeze -f /xfs_raid_path"));
...
ec2-create-snapshot -
...
bxc_exec("/usr/bin/ssh remote_xfs_freezer@hostname".escapeshellarg("sudo /usr/sbin/xfs_freeze -u /xfs_raid_path"));
...
ec2-create-volume --snapshot id_of_snapshot
...
ec2-attach-volume volume_id -i instance_id -d device
...
mdadm --assemble /dev/mdN /dev/sd[az]
MySQL?
Although you can one- handedly reset MySQL data to disk: FLUSH TABLES WITH READ LOCK , freeze the file system with your left foot: xfs_freeze -f / path, and press Enter with your nose to perform a snapshot API removal call: ec2-create-snapshot ... for now arranges asynchronous replication of MySQL to another DC :-). Interesting is the creation of base snapshots using xtrabackup - without the above described dances with tambourines.
You can back up the whole car at once!
If you separately backup disks, disk groups in raids, then, if there is a config, the process works stably and transparently. However, if you sometimes need (for example, a lightning struck the DC at the weekend :-)) to move between AZ (data centers) Amazon - it is easier to enter a level of abstraction above those of instances, snapshots and disks and operate with categories of roles and images (AMI - Amazon Machine Image ) machines. Through the API, you can assign tags to Amazon objects and filter samples by them.
We defined the roles: web machine, DBMS, balancer, monitoring machine and our backup scripts work like this:
- Find a car with the “DBMS” tag and get its ID
- “Freeze” car drives
- Make snapshots of disks. Set the tag "DBMS".
- "Unfreeze" machine discs
- To clear outdated snapshots, you can make a selection by tag and time and remove unnecessary.
However, it is much easier to back up a running virtual machine into an image (AMI image). At the same time, its settings and paths to the created snapshots are saved in the image, from which the machine disks will be created, which are attached to it at the start:
ec2-create-image id_of_instance [--no-reboot]
...
ec2-run-instances ami_image_id
Feel the difference! I create an image from a machine with a raid of 10 discs - one team. And I don’t bother with its discs and snapshots — all the information necessary for recreating a car is recorded into the AMI image automatically. The only drawback is that it is unclear how long it takes to drive disks / raids of a running machine in order to create an AMI image from it. Experienced to find that 2-5 seconds are enough - if less, then the disks / raids begin to recover when creating a new car. It is also very convenient that you can raise the machine from the backup to the image (AMI) in another data center (AZ).
Now we are creating a backup twice a day to the AMI image of all the machines in our cluster configuration (hundreds of gigabytes). Cleaning up "obsolete" images, along with their snapshots, is done by the script:
function bxc_delete_old_instance_backups( AmazonEC2 $ec2, $instanceRole = '', $bkpsMaxCount = 3) { if (!$instanceRole || !is_object($ec2) || $bkpsMaxCount<1 ) { bxc_log_error("Bad input params: ".__FUNCTION__); return false; } $response = $ec2->describe_images(array( 'Filter' => array( array('Name' => 'tag:Role', 'Value' => 'Image:'.$instanceRole), array('Name' => 'state', 'Value' => 'available'), ) )); if ( !$response->isOK() ) { bxc_log_amazon_error($response, __FUNCTION__.":".__LINE__); return false; } if ( isset($response->body->imagesSet->item) ) { $images = array(); $snaps = array(); foreach ( $response->body->imagesSet->item as $image ) { $images[ (string) $image->name ] = (string) $image->imageId; if ( isset($image->blockDeviceMapping->item) ) { foreach ( $image->blockDeviceMapping->item as $disk ) { $snaps[ (string) $image->name ][] = $disk->ebs->snapshotId; } } } if ($images) { krsort($images); } else { bxc_log("No images with tag:Role=Image:$instanceRole"); return true; } if ( count($images) <= $bkpsMaxCount ) { bxc_log("Nothing to delete"); return true; } $imagesToDelete = array_slice($images, $bkpsMaxCount, count($images)-$bkpsMaxCount, true); foreach ($imagesToDelete as $imageName=>$imageId ) { $r = $ec2->deregister_image($imageId); if ( $r->isOK() ) { bxc_log("Image deleted ok: ".$imageId); sleep(5);//Important. Wait to ami really be deleted ;-) foreach ( $snaps[$imageName] as $snapId ) { $rr = $ec2->delete_snapshot( $snapId ); if ( $rr->isOK() ) { bxc_log("Snapshot deleted ok: ".$snapId); } else { bxc_log_amazon_error($rr, __FUNCTION__); bxc_log_error("Cannot delete snapshot: ".$snapId); } } } else { bxc_log_amazon_error($r, __FUNCTION__); bxc_log_error("Cannot delete image: ".$imageId); } } return true; } else { bxc_log("No images with tag:Role=Image:$instanceRole"); return true; } return false; } Admin, Console or API
Amazon AWS has a nice admin area, but many things can only be done through the API. For Linux, you can install command line tools for working with Amazon APIs, which are fairly well documented. We use such tools:
ls /opt/aws/
AutoScaling-1.0.39.0
ElasticLoadBalancing-1.0.14.3
ec2-api-tools-1.4.4.1
env-set.sh
The last script sets the environment variables for the operation of the tools. Please note that almost every Amazon web service has its own set of utilities that need to be downloaded and installed separately. For managing machines, for example, the documentation is here , and command line utilities are here .
So far, the official SDK for working with the Amazon API for PHP shows itself well (the above script uses this particular library).
findings
You can work efficiently with Amazon EBS disks by combining them into a software raid. Unfortunately, you cannot mount an EBS disk to multiple virtual machines to work with a cluster file system of the type OCFS2, GFS2. Backups of both disks and EBS raids are made easy, you only need to understand the basic principles and get around the pitfalls. Conveniently backing up the machines as a whole, rather than on a separate disk - this will allow you to quickly deploy the configuration in a nearby data center if a lightning strikes or a plane crashes into your DC. Good luck to all!
Source: https://habr.com/ru/post/127932/
All Articles