Backup - virtual clones against non-consistent centaurs

or easy way to create consistent backups without stopping the server using virtual machine cloning

Perfect backup in vacuum

The system administrator, setting up backup data on the server, draws in his mind beautiful images. The backup script in good faith adds the data to the archive, where they lie intact, inspire peace of mind. A cataclysm happens, as a result of which the information on the disks turns into a dreary sequence of zeros without a single unit. Panic is growing, the director is locked in his office with a gun. And then a hero appears, coolly recovers data from the last backup, and after half an hour the server works as if nothing had happened. Under the solemn music, the hero goes into the sunset.



Rough reality makes corrections: if a lot of trifles are not foreseen when setting up a copy, then during recovery it may happen that some of the data in the backup will be corrupted in an incomprehensible way. Easy recovery will turn into a painful search for pieces in different archives and collecting one of them. Leaving into the sunset is delayed due to the consistency of the copy.

')

Copy inconsistency

The concept of inconsistency of a copy means that instead of a single set of data reflecting the state of the original at one time, the copy consists of several parts reflecting the state of the corresponding parts of the original at different points in time.



For example, if we simultaneously increase the same counter in two different files, then in the inconsistent copy these files may have a different value. Another example, if a man with a horse is walking along the road and we decided to copy them, then in the inconsistent copy there will be one centaur instead of them.

How do centaurs appear

The simplest and most popular backup method is file-based file system copying. The archive can save a full copy created by tar or cpio, or incremental with dump, rsync, bacula. All files in the file system are alternately bypassed, possibly checked for compliance with certain rules and copied to the archive.



The first obvious reason for the violation of consistency is a file change that occurs during copying. The server continues to perform its normal tasks, files are created, updated, deleted. The longer the copying takes, the larger the volume of all the data, the greater the rate of change of the data in the original - the greater the inconsistency of the copy. In some cases, it will be uncritically - for example, if a corporate website is copied at night, on which only the logs change at this time. But if changes to the SVN repository were being recorded at this time, then despite the fact that SVN uses transactions, after recovering from such a backup, you may encounter loss or mixing of versions of arbitrary components.

How clones are made

To solve this problem, there is a universal scheme - cloning a disk or file system, snapshots. Its implementations can be different - for example, with the means of the file system itself in UFS and ZFS, or a lower level, on a block device in LVM / DeviceMapper or VHD. From the user's side, it looks as if at the required moment the operating system creates a second copy of the file system or block device. It takes very little time to create a snapshot (from a few milliseconds to seconds), so without any prejudice to the operation of applications, all write operations are blocked. During the creation of the snapshot, no changes are made and the copy is consistent.



Are snapshots suitable for backup? For short-term needs, yes. For example, before any risky updates it will be useful to make a snapshot and, if something goes wrong, roll back the state of this snapshot, and if everything ends well, then the snapshot can be deleted. Using snapshots has its price - slowing down disk operations, increasing the consumption of disk space. Therefore, long-term use of snapshots should bring more benefits than performance will be lost. Snapshots are not suitable for backups because they are located in the same place as the original. If the original suffers, then a copy is likely to suffer.



In backup, snapshots are used as a fixed file system that can be copied to remote storage for an arbitrarily long time and the copy consistency at the end of the copy will not be violated. Copying can be done with the same dump / tar / rsync / bacula. After the snapshot is used, it is deleted so as not to waste resources.

Server data consistency

Does this solve the problem of clean recovery? Only partially. We receive a consistent copy of the file system, but do not receive a consistent copy of all server information. After all, there are applications that operate on data in RAM and save them to disk. In memory, these applications are consistent, but written to disk is not necessary. Information is written to the file in the order in which it is convenient for the application and the operating system. The point of view of the reservation system about this order is not interested in the application. The copy status corresponds to the state of the server disk after a sudden shutdown, for example, de-energizing.



For web projects, the potential victim of inconsistency is the MySQL database. Suppose we have an active project with a large number of data updates, data is constantly recorded in tables. The database server keeps data tables in RAM, periodically flushing changes to disk. An operating system is also attached to this, which can delay data in the write buffers, based on its own considerations about what is needed to improve performance. If at this moment we make a consistent copy of the file system, then there may be data in different tables that are relevant at different points in time, and in some tables the data integrity may be compromised if they were transferred to another place at that moment.



This leads us to the need to take into account the peculiarities of each application that changes data on the server. And adapt the backup process, forcing each application that changes data, save it to disk before creating a backup and refrain from changing it during copying. For the MySQL server, it consists in locking tables for writing and flushing changes to disk with a single FLUSH TABLES WITH READ LOCK command, from unlocking the tables after creating a copy with the UNLOCK TABLES command. And if this is a Java-application with third-party components, about the features of which nothing is known, the situation is complicated and only the complete stop of the application remains. For Windows applications, a special Quiesce mechanism is proposed, with the help of which the operating system informs applications that it is starting a backup and they need to flush their data to disk or otherwise bring them to disk in a consistent state.



In practice, it is this part that is either often overlooked when setting up a backup, or the sysadmin (or developer) deliberately refuses additional settings if he believes that the risk in his situation is insignificant and it is not worth the additional cost of complicating the copy script.

Virtual Clones of Virtual Machines

Virtual machines in recent years have become familiar both on the desktop and on the server. Progress makes people's lives more comfortable. It makes life more comfortable for system administrators. Server virtual machines are conveniently used for hardware abstraction, for efficient and dense use of hardware resources, for convenient administration of a large number of functionally different servers combined on a small number of physical servers. And virtualization allows you to make a perfect backup, without mythical creatures and fantastic mutants.



Many virtualization environments allow you to save a snapshot of a running virtual machine (as opposed to a disk, the term checkpoint is often used for this, and not the snapshot — not to confuse a memory snapshot with a disk snapshot). Basically, they are designed for development and testing, for example, to save the state of a working server before upgrading, in order to roll back when the outcome fails. From a backup point of view, a snapshot of the memory and disk is a consistent copy of all server data. Thus, we have a tool for getting the perfect backup, and it needs to be properly applied.



In Citrix XenServer (including XenServer Free) and XCP, use the xe vm-checkpoint command to create a snapshot of a virtual machine with memory and disk. At the time of the snapshot, the virtual machine is suspended and continues after completion. The snapshot is ready and in storage, but in this form its value as a backup is small - as is the case with disk snapshots, what is stored with the original may suffer along with the original. To get a copy that can be stored anywhere, you need to use the xe vm-export command, which saves the memory and disk of the virtual machine to an xva file at the output.



Xen Hypervisor 4 in the form of xend + xm, when placing machine disks on LVM, allows you to do about the same. Standard command xm save -c (save with checkpoint) allows you to save a snapshot of the memory and then continue the work of the virtual machine. The disk image is not created at the same time, assuming that someone else will do it. The first solution that comes to mind is to pause the domain, make a disk snapshot, and only after that do a virtual machine snapshot. But it will not work, so that the domain can be saved, Xen requires that it be in working condition. There are several ways to solve this problem, as a result, the most convenient option turned out to be with a small modification of xend (you can take our reduced patch for Debian 6 with Xen 4.0 and, if necessary, modify it to suit your needs: www.truevds.ru/misc.xen-checkpoint -clone-patch ). To do this, it is enough to indicate in the code of the function responsible for creating the checkpoint that the configuration of the saved virtual machine uses the snapshots of the disks, not their originals, and before resuming the virtual machine to make these snapshots using the standard LVM method. After that we will have a file with a snapshot of the memory of the virtual machine and an LVM partition with a snapshot of the disk.



In addition to Xen-based virtualization systems, VMWare and Hyper-V offer similar capabilities. In their terminology, checkpoint is called snaphost.

Good clone - turned off clone

It is not very expensive to store in the archive full copies of the memory and disks of the virtual machine, if it is some small server with small data. If you try to use this scheme for something serious, say, a production server for a visited site with 8 GB of RAM and about 100 GB of data on a disk, transferring over the network and storing 108 GB for each copy will quickly make the backup price prohibitive.



The most convenient way to store a large number of backup copies is incremental backups. During the next copy, the archive does not record a complete copy of the file system, but only the changes that occurred after the previous copy. To do this, we need a consistent file system in which the server's consistent data resides.



There is a single server state in which the data on the disk is saved by applications as correctly as possible and does not change in RAM. This state is a correctly stopped server. All applications that somehow care about the safety of their data are appreciated and readily obey when the operating system asks them to complete their work — via stop scripts or by sending a SIGTERM signal. The file system of a server stopped in this way is the maximum consistent state of all server data.



We need a file system in this state. We have a clone - a snapshot of a virtual machine and a snapshot of a disk. We need to start the clone virtual machine and give its operating system a shutdown command (shutdown). In XenServer / XCP, you will need to convert the snapshot into a template with the command xe snapshot-copy and then start a new machine on this template. Xen Hypervisor just starts the machine via xm restore .



The clone should not be able to harm the original. As a disk, a clone uses its own snapshot (in xend / xm, by default, this is not the case, but is fixed by the patch mentioned earlier that saves a virtual machine snapshot) - the original disc is not available to the clone. But if the original worked with the network, then the clone will have a network interface with the same settings. When enabled, there will be a conflict between the clone and the original for the right to communicate with the outside world. Therefore, the network interface of a clone must be deactivated or isolated within a separate virtual network that does not go to the real one. The server stopping method depends on how it is configured and most often the xm shutdow command is enough for the PV machine, and ACPI power off for the HVM.



After shutting down, the clone virtual machine can be deleted, and we still have a snapshot of the disk with a clean stop. This will be a file system with the purest consistency. With peace of mind, it can be sent to the archive in the same way that ordinary disk snapshots are copied.

Win price

There are minor difficulties associated with the fact that to run the clones you need RAM, which suddenly may not be enough; machine operation is suspended while memory is being saved. Their decision depends a lot on specific circumstances. For example, if we, as decent people, make a reservation at night, then we can painlessly take away half of the memory from the original and start the clone on the released half. A small modification of xend allows you to take a snapshot of memory even without suspending the virtual machine (in XCP, this is also technically possible, most likely the developers just haven't gotten to their hands).



Transferring servers to virtual machines for a high-load Internet project may be superfluous. Having dozens or hundreds of servers of the same type, it will be more practical to debug to the smallest detail and to unify the backup procedure by means of applications, then implementing it one many times than spending a small percentage of resources for virtualization overhead.



For smaller-scale Internet projects or for a heterogeneous set of corporate servers, the cost of organizing a pure backup using virtualization will be the lowest in most cases. As a result, we are able to make normal consistent server backups without adjusting backup scripts for all the variety of applications. This reduces the risk that some of the components will be forgotten or will not work. The hero, having restored the server in ten minutes, will jump into the sunset on a horse, without the risk of becoming a centaur.