Case analysis about backup verification (SureBackup) and using vSAN
Hi, Veeam Support is with you again! Today, a very interesting case is in the center of our attention: the SureBackup verification task ends midway through the snapshot creation, producing a general error message: “The specified feature is not supported by this version.” (This functionality is not supported in this version ).
The verification task included several virtual machines, but it was not possible to process a single virtual machine. Moreover, the verification task was based on the backup task, which worked without any errors, performing a backup and this VM itself.
Read about our investigation under the cut.
In fact, such errors can occur for several reasons - for example, there is a number of functionality limitations in the free hypervisor Free ESXi (therefore we do not support it). But this was not the case - the hypervisor was clearly not Free, all other machines were handled perfectly by the SureBackup task, and the problematic machine was normally backed up by the backup task.
However, since this machine became a stumbling block for the SureBackup task, it made sense to consider it more closely.
')
The first thing that caught my eye - this machine, let's call it VM01, was significantly larger than the others: one of its disks was larger than 2TB. Noting this fact, we began to analyze the job log of the job Veeam Backup & Replication. This is what was found there:
Honestly, at that moment logging did not help us much, but at least it became clear that the error message generator is VMware vSphere API.
Here, without further ado, we performed a search query by entering the source error message in it:
The record found contained the name of the disc. It turned out to be the same VMDK disk larger than 2TB, which has already attracted attention. Also included is the type of disk - seSparse , that is, a space-efficient sparse .
Here we remembered that when creating a snapshot disk larger than 2TB for a delta disk, VMware vSphere uses not VMFS (vmfssparse), but the seSparse format. This is written in the article Knowledge Base VMware KB 2058287 .
Our problem is probably somehow related to the use of a disk like seSparse for storing a delta (redo log). But what exactly was the root cause? And here we paid attention to one more detail: the initial VM and the virtual laboratory involved in the SureBackup task were located on the vSAN datastore.
It is well known that the SureBackup verification technology uses the Veeam vPower NFS datastore to “lift” virtual machines from it, while the deltas (redo logs) for the machines in the verification task are redirected to the main datastore (in our case, it is vSAN):
That is, such a storage system was formed in which the main disks were located on the NFS datastore, and the deltas (redo logs) - on the vSAN. But since for all machines except the VM01, this system worked fine, we came to the conclusion that the issue of compatibility between seSparse and vSAN should be investigated.
The investigation led us to the VMware vSAN document , which states: “Virtual SAN does not support SE Sparse disks.” —The virtual SAN does not support SE Sparse disks .
The document, however, was related to vSphere v5.5, and in our case it was vSphere 6.0. We contacted colleagues from VMware who confirmed that this restriction holds for both version 6.0 and version 6.5.
Finally, all the pieces of the puzzle were put together, and here's the result: a storage system configuration is impossible where the main disk would be on NFS / VMFS, and the delta file (redo log) seSparse on vSAN - because, as stated in the VMware documentation , snapshot inherits the VMFS_type type.
The attentive reader will ask a reasonable question: why did a large backup (more than 2TB) normally work for backup? After all, seSparse is not supported by vSAN? The answer is simple: when creating a VM snapshot whose main disk is located on vSAN, VMware uses VSANSPARSE to save the delta.
Note: You can read more about VSANSPARSE in the VMware article ( downloadable PDF ).
Admin note:
We advised the user to transfer the virtual lab sandbox to the datastore without using vSAN. In this case, when redirecting the type of snapshot will not change.
Similar problems can arise when working with VM on vSAN, if you try to start Instant Recovery for machines with disks of large size (> 2TB). Note that the above solution is applicable here, since the snapshot is also redirected.
Further studies of the question showed that in some cases the problem manifests itself also when operating in the hot-add mode (virtual appliance). In any case, you need to carefully analyze which datastores and which types of snapshots will be involved in a particular operation. Well, if you still have questions - we are always ready to help.
On the eve of the winter holidays, Veeam gives presents again: we are playing 6 passes at the 2018 VeeamON conference, which in the new year will be held in different regions of the world. The ticket includes a ticket on both sides, accommodation in a comfortable department near the conference venue, and, of course, attendance at all conference events (including those for which a reservation is required).
All you need to do to take part in the draw is to register here .
If you are reading us, being, for example, in Australia or Canada, then you can open the page of the desired regional conference, links to them here .
Registration will close on December 31, 2017, so it’s best to hurry.
The names of the six lucky ones will be announced in the new, 2018. You can follow the news on our page in FB , as well as on Twitter .
The verification task included several virtual machines, but it was not possible to process a single virtual machine. Moreover, the verification task was based on the backup task, which worked without any errors, performing a backup and this VM itself.
Read about our investigation under the cut.
In focus
In fact, such errors can occur for several reasons - for example, there is a number of functionality limitations in the free hypervisor Free ESXi (therefore we do not support it). But this was not the case - the hypervisor was clearly not Free, all other machines were handled perfectly by the SureBackup task, and the problematic machine was normally backed up by the backup task.
However, since this machine became a stumbling block for the SureBackup task, it made sense to consider it more closely.
')
The first thing that caught my eye - this machine, let's call it VM01, was significantly larger than the others: one of its disks was larger than 2TB. Noting this fact, we began to analyze the job log of the job Veeam Backup & Replication. This is what was found there:
Honestly, at that moment logging did not help us much, but at least it became clear that the error message generator is VMware vSphere API.
Analyzing VMware Logs
Here, without further ado, we performed a search query by entering the source error message in it:
The record found contained the name of the disc. It turned out to be the same VMDK disk larger than 2TB, which has already attracted attention. Also included is the type of disk - seSparse , that is, a space-efficient sparse .
Here we remembered that when creating a snapshot disk larger than 2TB for a delta disk, VMware vSphere uses not VMFS (vmfssparse), but the seSparse format. This is written in the article Knowledge Base VMware KB 2058287 .
Our problem is probably somehow related to the use of a disk like seSparse for storing a delta (redo log). But what exactly was the root cause? And here we paid attention to one more detail: the initial VM and the virtual laboratory involved in the SureBackup task were located on the vSAN datastore.
It is well known that the SureBackup verification technology uses the Veeam vPower NFS datastore to “lift” virtual machines from it, while the deltas (redo logs) for the machines in the verification task are redirected to the main datastore (in our case, it is vSAN):
That is, such a storage system was formed in which the main disks were located on the NFS datastore, and the deltas (redo logs) - on the vSAN. But since for all machines except the VM01, this system worked fine, we came to the conclusion that the issue of compatibility between seSparse and vSAN should be investigated.
The investigation led us to the VMware vSAN document , which states: “Virtual SAN does not support SE Sparse disks.” —The virtual SAN does not support SE Sparse disks .
The document, however, was related to vSphere v5.5, and in our case it was vSphere 6.0. We contacted colleagues from VMware who confirmed that this restriction holds for both version 6.0 and version 6.5.
Finally, all the pieces of the puzzle were put together, and here's the result: a storage system configuration is impossible where the main disk would be on NFS / VMFS, and the delta file (redo log) seSparse on vSAN - because, as stated in the VMware documentation , snapshot inherits the VMFS_type type.
The attentive reader will ask a reasonable question: why did a large backup (more than 2TB) normally work for backup? After all, seSparse is not supported by vSAN? The answer is simple: when creating a VM snapshot whose main disk is located on vSAN, VMware uses VSANSPARSE to save the delta.
Note: You can read more about VSANSPARSE in the VMware article ( downloadable PDF ).
Admin note:
- VSANSPARSE snapshots are created on vSAN drives.
- VMFS_sparse or seSparse snapshots are created on regular VMFS disks, and the size of the disk and the VMFS version matter (for example, snapshots on VMFS6 will always have seSparse type regardless of size).
- In the case of redirecting to another datastore, the type of disk snapshot will be inherited from the parent disk.
We offer a solution
We advised the user to transfer the virtual lab sandbox to the datastore without using vSAN. In this case, when redirecting the type of snapshot will not change.
Similar problems can arise when working with VM on vSAN, if you try to start Instant Recovery for machines with disks of large size (> 2TB). Note that the above solution is applicable here, since the snapshot is also redirected.
Further studies of the question showed that in some cases the problem manifests itself also when operating in the hot-add mode (virtual appliance). In any case, you need to carefully analyze which datastores and which types of snapshots will be involved in a particular operation. Well, if you still have questions - we are always ready to help.
In addition to the useful, here is also a pleasant
On the eve of the winter holidays, Veeam gives presents again: we are playing 6 passes at the 2018 VeeamON conference, which in the new year will be held in different regions of the world. The ticket includes a ticket on both sides, accommodation in a comfortable department near the conference venue, and, of course, attendance at all conference events (including those for which a reservation is required).
All you need to do to take part in the draw is to register here .
If you are reading us, being, for example, in Australia or Canada, then you can open the page of the desired regional conference, links to them here .
Registration will close on December 31, 2017, so it’s best to hurry.
The names of the six lucky ones will be announced in the new, 2018. You can follow the news on our page in FB , as well as on Twitter .
Additional materials:
Source: https://habr.com/ru/post/344524/
All Articles