Today, again, I am pleased to present you with useful tips from my colleague Yevgeny Ivanov, Team Lead of the Veeam technical support team. This time, Zhenya shared recommendations for working with backups and replicas. I hope they will help you avoid common mistakes, and your replicas and backups will never be the “weak link” in the recovery process, if needed.

So, welcome under cat.



In my previous article, we figured out how to optimize the load on the components of the backup infrastructure, and looked at typical configuration errors. We turn to another important topic - the proper preparation and implementation of recovery. We will also analyze it with real examples with which the technical support team has worked.
')

Backup without restaura - money to the wind

We are regularly approached by users who find themselves in similar difficult situations: you need to perform a recovery from the backup, but when you try to do this, people stumble upon an unsolvable problem for them. And this problem is not a lack of backup, CryptoLocker activity or something like that. This is “only” insufficient attention to checking backups and replicas for recoverability. Many people often focus solely on the process of creating a backup, forgetting that simply having a backup copy is not a panacea for possible troubles. It is necessary to understand that recovery is a completely different process, which has its own characteristics, and which must be controlled and tested before launching in production. Here are some illustrative examples:

1. The user has a failure in the operation of a critical virtual machine with a size of 20 TB. Downtime, of course, is unacceptable, and the admin starts the instant recovery process (VM instant recovery) - after 5 minutes the machine is raised. But we remember that such a state of the machine can be involved only temporarily - it must be migrated to the production datastore. And in this example, as it turned out, the infrastructure capabilities did not allow copying 20 TB of data in a reasonable time. In the settings of the instant recovery process, it was chosen to save the changes to the C drive : Veeam Backup & Replication server (as opposed to vSphere) - as a result, of course, the free disk space was quickly filled. By the time the user turned in support, the VM had changes that could not be ignored. That is, we have a situation when it is impossible to quickly finalize the process of instant recovery of a critical machine - how can we save the data
   
   Frankly, I don’t remember all the details of the final, but I remember that in the end we didn’t think of anything brilliant. Clients on their side somehow solved this problem by expanding the C drive: from the reserves, copied the most important files and then turned off the VM and so migrated. In general, the miracle did not happen.
2. The user had one domain controller in the infrastructure, and all components of Veeam Backup & Replication were configured using DNS. Yes, yes, that's right, you heard right. There were hundreds of options for the development of events, not less, but the reality went like this: people planned maintenance and decided to switch to a replica of their domain controller. They involved a scheduled switch, which, in general, is recommended to do in such situations. At the first stage, everything went fine, and at the second stage, the initial VM was turned off for a short time in order to transfer the remaining data. Of course, the switch task immediately ended with an error, because the DNS has stopped working.
   
   Fortunately, here we managed to cope with the situation by turning on the replica manually from the vSphere (in general, this operation is not recommended to be performed independently, as you will see from the following example). But, as you understand, the maintenance process was interrupted and postponed. In addition, we had to manually add the host names to the file C: \ Windows \ System32 \ drivers \ etc \ hosts on the Veeam Backup & Replication server to ensure correctness when switching back.
3. At one client, the entire backup infrastructure was built around tape drives, and only short chains of files were stored on the disk. When they needed to restore a number of files from a large file server, it turned out that no machine could be used as an auxiliary repository when restoring from tape, since none of them had enough free space. (You can read about recovery from a magnetic tape directly and using an auxiliary repository here (for the time being in English)).

I think that in all three examples, users, so to speak, were in the grip of illusions - they assumed that if the backup was successful, then there would be no problems with the restoration. But this, as you understand, is by no means always the case, and therefore it is necessary to prepare for recovery as carefully as for backup. To get started is to study the user manual , which contains fairly detailed information on different types of recovery. At the beginning of each paragraph lists the requirements, preparatory actions and possible limitations. The description of recovery from magnetic tapes or from hardware storage images can be found in the documentation sections and in our articles on Habré. In addition, steps to prepare the restoration of application objects using the tools of Veeam Explorers are described in the “Planning and preparation” section of the planning guide for each of the tools. I recommend that you read them carefully - this will help you properly prepare the system for recovery if necessary. In Russian, instructions for recovering a SQL Server database are provided here .

Why not work with replicas from the vSphere console?

The idea is that Veeam replicas are ordinary virtual machines that it would seem logical to work with using the vSphere toolkit, in particular, the vSphere client. However, we do not recommend doing this, and here's why: switching to a replica in Veeam Backup & Replication is a rather complicated process that requires strictly sequential steps (so that if something happens, you can roll back a step) and correct final actions - just look at picture illustrating the process:



If you decide to include a replica from the vSphere client, then in the future you will most likely face a number of problems:

1. The mechanism for switching to a replica from Veeam Backup & replication (shown in the diagram) will no longer work for this machine.
2. The data in the Veeam Backup database will not correspond to the real state of the VM. In the worst case, the repair will have to edit the database.
3. Even data loss is possible, as in this example: the user manually included the replica in the vSphere client and decided to continue working with it. After some time, he noticed that the replica is still displayed in the console Veeam Backup & Replication, and decided to remove it as useless. I clicked on it with the right button and gave the command “Delete from disk” . Veeam Backup & Replication immediately deleted the replica from the disk, which, for a minute, was already used as a normal VM and contained the necessary and useful data.

Of course, there are situations when you still have to turn on the replica from the vSphere client - as a rule, these are cases when the Veeam server is turned off and the replica needs to be turned on with a delay. But if everything is in order with the Veeam server, then you need to work with replicas from its console.

Also, do not delete replicas using vSphere client. Veeam Backup & Replication will remain in the dark about such a change, and this is fraught with errors and outdated data. If you no longer need a replica, delete it using the Veeam console, and not as a VM from the vSphere client. So you will always have a current list of replicas.

"O" - caution, updates!

Here we mean, of course, updates for hypervisors and various applications that are backed up with Veeam. If you look at them from the point of view of working with Veeam Backup & Replication, the updates can be divided into 2 categories: large, serious, introducing a lot of changes - and small.

Consider first the first category.

The most important updates are those intended for the hypervisor. Before you install such an update, you must make sure that it is supported by Veeam Backup & Replication. Such updates introduce many changes to the libraries and APIs that Veeam Backup & Replication uses, so in order to officially declare their support, you need to update the code of Veeam Backup & Replication and conduct thorough testing.

It should also be borne in mind that, for example, VMware does not provide preliminary access to the latest vSphere versions for software manufacturers, so Veeam developers and testers receive a new version simultaneously with the rest of progressive humanity - therefore, between the release of VMware and officially announced support, certain time. The number and variety of changes necessary for making is such that there is little chance of finding room for their simple hotfix - and official support is usually announced along with the release of the release version of Veeam Backup & Replication.

As a result, there is an awkward moment when, after the release of the new version of vSphere, the number of requests for technical support increases dramatically, because users rush headlong to install a new version, and their backups, of course, immediately stop working immediately. We - technical support Veeam - have to explain to users what exactly they did wrong, ask them to roll back (if possible) or invent intricate ways to break the deadlock. Therefore, before installing a serious update, be sure to check its compatibility with the software that works for you, I beg you!

All of the above applies to applications that you back up and expect to restore using Veeam. The Veeam Explorers toolbar also has a list of supported versions of the corresponding applications, which is updated with each release of Veeam Backup & Replication. Therefore, before installing your new version of your application - whether it be Exchange, Oracle or SharePoint - be sure to re-read the relevant section of the Veeam Explorers documentation .

To the second category, i.e. small updates include, for example, new versions of VMware Tools, cumulative updates from Exchange, vSphere security updates, etc. As a rule, they do not carry with them any serious modifications, and in most cases Veeam Backup & Replication does not have problems with them. (Therefore, for them, there are no public announcements of official support in the product.) However, in our practice, there were cases when such updates so significantly changed the usual course of things that led to errors in the work of Veeam Backup & Replication. In such situations, after confirming the problem, Veeam engineers are trying to quickly release a hotfix.


As you understand, patches and updates can lead to problems not only with backups, but also with the applications for which these backups are made. And here virtual labs will help you - Veeam DataLabs. You have probably heard about the functionality of SureBackup, designed for verification of backups. It is based on the use of DataLabs, with the creation of an isolated environment in which you can, in particular, test updates before installing them in production. I strongly advise you to do so - save yourself a lot of nerve cells. And if someone else does not know about SureBackup, I recommend reading the documentation .

Perhaps today I have everything, thank you for your attention!

What else to read

Articles on Habré:

• The case of Pixar or again about the importance of testing backups
• Features long-term backup storage
• Best Practices for Backup Policy
• Recommendations for backup and recovery after the End of the World policy

User Guide (in Russian)