System backup of virtual machines or how to save money organization
Virtualization is a very convenient and competent solution for the work of a modern enterprise with a multitude of tasks. It allows you to distribute tasks between different servers, and their administration is divided among several employees who are responsible for each server. Thus, a distributed IT infrastructure of an enterprise is created, but concentrated in one or several physical servers — virtualization helps to more intelligently distribute the resources of modern multiprocessor systems — by placing several virtual machines on a single physical server.
A virtual machine is represented by several files that are located on the host's hard disk, which allows you to perform actions such as copying, deleting, and restoring on it. Everything is simple and clear: there are some files (configuration and hard disks) they run in a certain hypervisor, and everything works like a real server. It really saves the company money and time for the admin. In particular, a good solution is a cluster, several hosts are combined here, and if one virtual machine fails, they switch to another.
In our organization, we use a Windows Server 2012 R2 Data Center-based failover cluster (Hyper-V cluster). From a little experience, I can say that the cluster is quite a handy thing, since there is a possibility of releasing each host for any administrative tasks by simply transferring - migrating virtual machines to another host - for example, installing updates or software, plus the fact that If one of the hosts “burns”, then the virtual machines will continue to work with almost no pause - the migration will work (“the main thing is to work”) - everything will continue to work, but if it does not work and the virtual machine is damaged, then the data can be pulled out Directly from a hard disk file.
Most of the problems that we had were damage or disappearance of xml files, which are configuration files or part of snapshots information. This occurred, for the most part, when the host rebooted, when the virtual machine did not migrate to another cluster host (all the problematic machines were 2008 servers); the place on the virtual hard disk ended and for some reason the snapshot disappeared; after removing the snapshot, no merging took place, and the snapshot was deleted. What are the problems and with which tambourine to approach them - every time there was no desire to find out. Therefore, we took the following steps to optimize the system:
')
Our virtual machine park began to occupy less disk space, and the machines were divided according to tasks and, accordingly, they were assigned the necessary amount of resources.
All this was a preparation for creating a backup system for virtual machines. We started to consider existing ready-made systems, but we faced the main problem - this is money. Yes, as always they! We were told: “There is money, but they are few ...” Which means “We will not give anything!”. But still, we did not lose hope for funding, so we began to consider first paid options - the first was Microsoft's DPM. Read about him - there are a lot of good reviews, but there are many problems.
We tried to install, tried to install several times: the first time MS SQL died it is not clear why, the second time, the most epic outcome that one could expect - DPM killed itself! We created 2 virtual machines: 1 - sql-server, 2 - dpm with attached physical disk (the machines were not located in the cluster, but on a separate Hyper-v Rezerv server), installed agents in the cluster and started testing, as a result, everything works - all backups . They started testing the recovery functions - they restored it to the Hyper-v Rezerv server, but they saw the words “the Sql-server virtual machine will be deleted”. It turned out that the id of the machine that was restored coincided with the id of the sql-server machine, because we tested the recovery on a reference virtual machine, which we copy by export and so we create new virtual machines, but this time the sql-server was created by registering in the place ... So Thus, we got two virtual machines with the same id - one reference in the cluster, the other on the backup Hyper-V Rezerv host. When restoring a virtual machine, DPM "looks" at id, and if it matches, it deletes the virtual machine, and restores the copy to its place. It turned out very evil and ridiculous mistake. I remembered forever that situations should not be allowed so that in a virtualization system, even on different hosts, clusters, and everywhere there were machines with the same id.
I thought: “Damn it, now I'll create it all over again.” But it turned out that with all subsequent installations from different distributions, even on different hosts, the same unknown error popped up during the installation process. On the forums they wrote: broken distribution. So after all, yesterday everything was fine! Yes, and downloaded a few more times ... In general, magic, the evil eye, or just melkosoft ... History is tragic. Therefore, spat on small-soft and began to look the other way.
DPM is a pretty good thing and not expensive, but it uses MS SQL-Server Standart, it stores backups on a dynamic disk - why I focus on this, because it is not convenient: SQL-Server costs money, is complex, and without its bases it works with dynamic the disk becomes useless, that is, the base is damaged and everything ... All backups are lost ... Or you need a very strong tambourine ... It is not clear why not to use a static disk, which could be connected anywhere with ease, rather than dynamic. And DPM did not work with tape storage: it constantly inventoried tapes, but did not record anything.
The next product that you definitely liked is Veeam. It is very convenient and its green color certainly pleases the gray admin interface, but the price for organizing is stunning (we have a cluster of 6 dual-processor blades). We tested it and would be ready to use if you have money. But alas.
In the process of reviewing products for backups, we backed up manually, then using a script on PowerShell. And later they decided to finish the script, which would fulfill the tasks.
Some will say that in vain ... What why reinvent the wheel ... What is a decent free program.
I chose the script because it does not need a database server. All you need is a “what when to backup” sign and logs for debugging. And, in fact, it is free and this self-development.
But there is one significant drawback: with the help of paid and free programs, you can extract data - files or databases directly from a virtual machine, and the script allows you to make a copy of only the entire virtual machine. But we are working on it and maybe in the following articles I will demonstrate it. I am still a beginner - 2 years in administration and 3 years of working with juzverej, and you know, it is easier to communicate with servers.
So, we planned to backup completely virtual machines - why? Yes, because
What is critical for this method is time: no matter how fast your network is, building an exported copy over the network is a long time for Hyper-V - about 40 GB per hour on a gigabit grid - at least for us, if you have faster, please explain in the comments why. From this calculation for the night, from 00:00 to 7:00, we have the opportunity to save about 280 GB, but this is night and this is small-sized software, and the virtual machine of 300 GB is quite a large and cumbersome animal - therefore, after analyzing our tasks, we limited ourselves 100-120 GB in size for virtual computers. This range includes almost all of our servers, even database servers. We set up backup depending on the variability and importance of the virtual machine - the more often it changes, the more often we copy it, but this does not apply to the database server - we copy it once a week, and the database itself with SQL means - 4 times a day - thus, there is a complete copy a week ago, and the bases themselves everyday.
With file servers and such monsters as update servers (all machines are more than 200 GB) did the following: if physical disks are not connected to the virtual machine, then backup once every six months or once every three months, if connected, then backup once virtual machine configurations, because it is impossible to export a physical disk. In such machines, you will have to adjust the copying of files with a separate script or manually - we do this with the following batch file:
Drive H is a drive connected via iSCSI.
Thus, we distinguished file backups and backups of virtual machines. In addition, for greater reliability, we have demarcated networks: the Control network — for hosts and the enterprise network — for virtual machines. There is no access to the hosts from the enterprise network, which provides protection both from unauthorized access and from viruses that are successfully distributed by users. The Control network is physically separated — these are separate switches that are connected to individual ports of the hosts, and when configuring virtual switches, the hosts use only the control network. For the enterprise network, a separate virtual switch has been created that is not used by hosts. Also, problems in the enterprise network, for example, rings or the “enraged” network card, will not affect the performance of the Control network.
I will describe the configuration. Immediately clarify that the operating system on all servers - Windows Server 2012 R2 Data Center. The cluster consists of 6 Fujitsu BX924 S3 blades (3 pieces) and BX924 S4 (3 pieces), which are located in the BX900 S2 chassis. Through the FC switch blades are connected to the Eternus DX80 disk shelf, on which LUNs are allocated - 250 GB each for the operating system, 20 GB quorum and 6 TB for the virtual machine storage. Cluster - Hyper-V cluster. There is also a single-node Fujitsu RX200 server with the Hyper-V role that hosts virtual machines: the main controllers of the enterprise domain and the main DC cluster, the secondary ones are located in the cluster. Back in our server park, a Supermicro backup server with two processors and 196 GB of RAM is the same Hyper-V Rezerv server. It takes the role of Hyper-V and exports virtual machines to it - in case of loss of a virtual machine, you can restore it from backup (import it into Hyper-V by importing with registration in place, if necessary immediately) in just a couple of minutes.
The Control network is a single-gigabit network, a 10-gigabit enterprise network within servers and a 1-gigabit network in the remaining network segments.
We now turn to the script itself. I wrote it so that it could find a virtual machine in the cluster, connect to the cluster node, shut down or save the virtual machine and, in fact, export it to the ball on the Hyper-V REZERV, and then turn on the virtual machine.
Here is the script text:
The script itself is not complicated, it has a functional for tracking errors - try catch, there are a number of messages that are written to the log and make it clear how successful the export was. There is one snag - this script runs on behalf of the domain administrator, you must tick: "for users logged in" and "run with the highest rights." Without these checkboxes, a remote connection to another host will fail. You also need to configure Powershell to export a virtual machine from one remote computer to another orb - multi-hop authentication. set up for this article . In order for hyper-v to be able to export to a network folder, it is necessary to add the name of the host from which it is being exported and give full access in its security settings. On the backup server Hyper-V REZERV, a script is running that deletes files and folders that have been modified more than 63 days. It also has Veeam Backup Free Edition, which once a month records all the folders with virtual machines on the fiber channel on the fiber channel that they exported within a month.
To summarize: a script has been created for the complete export of a virtual machine - on the one hand, this is not convenient, on the other, it gives a chance to instantly restore full server functionality. A full copy is really long, but since the hosts are separated into a separate network that is small and physically separated from the enterprise network, this does not degrade the performance of the enterprise. Some machines need to be turned off before exporting, for example, such as a database server to get a valid backup, so I advise you to make such servers small, but if this does not work, then only backup databases, for example, using MS SQL using a schedule. For smaller virtual machines, it is better to use dynamic disks - so export will take a little time.
For example, all our domain controllers use dynamic disks, and therefore such virtual machines weigh up to 20 GB - export takes 30-40 minutes. Also, some machines can be backed up “on hot”, without putting the machine into a saved state or without completing the work, but I did not find detailed articles where such export tests were described, if you know more, therefore, please write in the comments. Also, all I need to do is to check the logs - whether the script has worked or not, so I believe that such a system can really provide the company with protection against data loss and save money.
PS: I am very grateful to the users of the Technet forum, without their help, building my first cluster and setting up backups would be much more difficult. I hope that this article so far an inexperienced admin will be useful and interesting!
A virtual machine is represented by several files that are located on the host's hard disk, which allows you to perform actions such as copying, deleting, and restoring on it. Everything is simple and clear: there are some files (configuration and hard disks) they run in a certain hypervisor, and everything works like a real server. It really saves the company money and time for the admin. In particular, a good solution is a cluster, several hosts are combined here, and if one virtual machine fails, they switch to another.
In our organization, we use a Windows Server 2012 R2 Data Center-based failover cluster (Hyper-V cluster). From a little experience, I can say that the cluster is quite a handy thing, since there is a possibility of releasing each host for any administrative tasks by simply transferring - migrating virtual machines to another host - for example, installing updates or software, plus the fact that If one of the hosts “burns”, then the virtual machines will continue to work with almost no pause - the migration will work (“the main thing is to work”) - everything will continue to work, but if it does not work and the virtual machine is damaged, then the data can be pulled out Directly from a hard disk file.
Most of the problems that we had were damage or disappearance of xml files, which are configuration files or part of snapshots information. This occurred, for the most part, when the host rebooted, when the virtual machine did not migrate to another cluster host (all the problematic machines were 2008 servers); the place on the virtual hard disk ended and for some reason the snapshot disappeared; after removing the snapshot, no merging took place, and the snapshot was deleted. What are the problems and with which tambourine to approach them - every time there was no desire to find out. Therefore, we took the following steps to optimize the system:
')
- transition of all virtual machines and hosts to 2012 R2 DATA CENTER, because it showed itself in practice quite stably and more quickly: it shuts down quickly, turns on and restarts, saves the budget through AVMA activation — unlimited activation of 2012 R2 DATA CENTER virtual machines inside the host;
- deleting all snapshots from virtual machines (due to snapshots, a virtual machine can add significantly in size);
- Reducing the size of static hard disks of virtual machines - allocating as much space as needed (for example, for database servers), especially if you can always increase it if necessary;
- use of dynamic hard disks for virtual machines that do not need fast disk speed (for example, domain controllers, license servers, etc.).
Our virtual machine park began to occupy less disk space, and the machines were divided according to tasks and, accordingly, they were assigned the necessary amount of resources.
All this was a preparation for creating a backup system for virtual machines. We started to consider existing ready-made systems, but we faced the main problem - this is money. Yes, as always they! We were told: “There is money, but they are few ...” Which means “We will not give anything!”. But still, we did not lose hope for funding, so we began to consider first paid options - the first was Microsoft's DPM. Read about him - there are a lot of good reviews, but there are many problems.
We tried to install, tried to install several times: the first time MS SQL died it is not clear why, the second time, the most epic outcome that one could expect - DPM killed itself! We created 2 virtual machines: 1 - sql-server, 2 - dpm with attached physical disk (the machines were not located in the cluster, but on a separate Hyper-v Rezerv server), installed agents in the cluster and started testing, as a result, everything works - all backups . They started testing the recovery functions - they restored it to the Hyper-v Rezerv server, but they saw the words “the Sql-server virtual machine will be deleted”. It turned out that the id of the machine that was restored coincided with the id of the sql-server machine, because we tested the recovery on a reference virtual machine, which we copy by export and so we create new virtual machines, but this time the sql-server was created by registering in the place ... So Thus, we got two virtual machines with the same id - one reference in the cluster, the other on the backup Hyper-V Rezerv host. When restoring a virtual machine, DPM "looks" at id, and if it matches, it deletes the virtual machine, and restores the copy to its place. It turned out very evil and ridiculous mistake. I remembered forever that situations should not be allowed so that in a virtualization system, even on different hosts, clusters, and everywhere there were machines with the same id.
I thought: “Damn it, now I'll create it all over again.” But it turned out that with all subsequent installations from different distributions, even on different hosts, the same unknown error popped up during the installation process. On the forums they wrote: broken distribution. So after all, yesterday everything was fine! Yes, and downloaded a few more times ... In general, magic, the evil eye, or just melkosoft ... History is tragic. Therefore, spat on small-soft and began to look the other way.
DPM is a pretty good thing and not expensive, but it uses MS SQL-Server Standart, it stores backups on a dynamic disk - why I focus on this, because it is not convenient: SQL-Server costs money, is complex, and without its bases it works with dynamic the disk becomes useless, that is, the base is damaged and everything ... All backups are lost ... Or you need a very strong tambourine ... It is not clear why not to use a static disk, which could be connected anywhere with ease, rather than dynamic. And DPM did not work with tape storage: it constantly inventoried tapes, but did not record anything.
The next product that you definitely liked is Veeam. It is very convenient and its green color certainly pleases the gray admin interface, but the price for organizing is stunning (we have a cluster of 6 dual-processor blades). We tested it and would be ready to use if you have money. But alas.
In the process of reviewing products for backups, we backed up manually, then using a script on PowerShell. And later they decided to finish the script, which would fulfill the tasks.
Some will say that in vain ... What why reinvent the wheel ... What is a decent free program.
I chose the script because it does not need a database server. All you need is a “what when to backup” sign and logs for debugging. And, in fact, it is free and this self-development.
But there is one significant drawback: with the help of paid and free programs, you can extract data - files or databases directly from a virtual machine, and the script allows you to make a copy of only the entire virtual machine. But we are working on it and maybe in the following articles I will demonstrate it. I am still a beginner - 2 years in administration and 3 years of working with juzverej, and you know, it is easier to communicate with servers.
So, we planned to backup completely virtual machines - why? Yes, because
- There is no point in backing up snapshots of virtual machines — which takes an hour or 2 to set up, for example, license servers, video conferencing, even terminals (if they are without data) —simply it is easier to save a fully configured machine and quickly tune in than to play a strategy with snepshots (from which it grows );
- If you use dynamic disks, many virtual machines without snepshots weigh up to 20 GB (for example, domain controllers) - this is an hour of export - that is, an hour and we have a complete working copy of a virtual machine, which we can deploy locally on a backup server 2 minutes, provided that it was exported there.
What is critical for this method is time: no matter how fast your network is, building an exported copy over the network is a long time for Hyper-V - about 40 GB per hour on a gigabit grid - at least for us, if you have faster, please explain in the comments why. From this calculation for the night, from 00:00 to 7:00, we have the opportunity to save about 280 GB, but this is night and this is small-sized software, and the virtual machine of 300 GB is quite a large and cumbersome animal - therefore, after analyzing our tasks, we limited ourselves 100-120 GB in size for virtual computers. This range includes almost all of our servers, even database servers. We set up backup depending on the variability and importance of the virtual machine - the more often it changes, the more often we copy it, but this does not apply to the database server - we copy it once a week, and the database itself with SQL means - 4 times a day - thus, there is a complete copy a week ago, and the bases themselves everyday.
With file servers and such monsters as update servers (all machines are more than 200 GB) did the following: if physical disks are not connected to the virtual machine, then backup once every six months or once every three months, if connected, then backup once virtual machine configurations, because it is impossible to export a physical disk. In such machines, you will have to adjust the copying of files with a separate script or manually - we do this with the following batch file:
xcopy F:\DATA H:\BackUP /H /Y /C /R /E Drive H is a drive connected via iSCSI.
Thus, we distinguished file backups and backups of virtual machines. In addition, for greater reliability, we have demarcated networks: the Control network — for hosts and the enterprise network — for virtual machines. There is no access to the hosts from the enterprise network, which provides protection both from unauthorized access and from viruses that are successfully distributed by users. The Control network is physically separated — these are separate switches that are connected to individual ports of the hosts, and when configuring virtual switches, the hosts use only the control network. For the enterprise network, a separate virtual switch has been created that is not used by hosts. Also, problems in the enterprise network, for example, rings or the “enraged” network card, will not affect the performance of the Control network.
I will describe the configuration. Immediately clarify that the operating system on all servers - Windows Server 2012 R2 Data Center. The cluster consists of 6 Fujitsu BX924 S3 blades (3 pieces) and BX924 S4 (3 pieces), which are located in the BX900 S2 chassis. Through the FC switch blades are connected to the Eternus DX80 disk shelf, on which LUNs are allocated - 250 GB each for the operating system, 20 GB quorum and 6 TB for the virtual machine storage. Cluster - Hyper-V cluster. There is also a single-node Fujitsu RX200 server with the Hyper-V role that hosts virtual machines: the main controllers of the enterprise domain and the main DC cluster, the secondary ones are located in the cluster. Back in our server park, a Supermicro backup server with two processors and 196 GB of RAM is the same Hyper-V Rezerv server. It takes the role of Hyper-V and exports virtual machines to it - in case of loss of a virtual machine, you can restore it from backup (import it into Hyper-V by importing with registration in place, if necessary immediately) in just a couple of minutes.
The Control network is a single-gigabit network, a 10-gigabit enterprise network within servers and a 1-gigabit network in the remaining network segments.
We now turn to the script itself. I wrote it so that it could find a virtual machine in the cluster, connect to the cluster node, shut down or save the virtual machine and, in fact, export it to the ball on the Hyper-V REZERV, and then turn on the virtual machine.
Here is the script text:
$LOG="C:\ClusterStorage\Volume1\ps\log\log.txt" // $h="Gse264-bl1","Gse264-bl2","Gse264-bl3","Gse264-bl4","Gse264-bl5","Gse264-bl6" // $VmName="Stream.local.ru (2012r2)" // $blade= hostname // , $password = ConvertTo-SecureString "Ofiget_Arbuz" -AsPlainText —Force // Control $cred= New-Object System.Management.Automation.PSCredential ("control", $password ) // foreach($i in $h){ // $vms=Get-ClusterNode $i | Get-Clusterresource| ?{$_.ResourceType -eq 'Virtual Machine'}|Get-Vm // foreach ($cn in $vms){ // if ($cn.name -eq $VmName) : {write "$(get-date -format "dd.MM.yy.HH.mm.ss") $VmName $i" | out-file $LOG —append // if ($i -eq $blade) // , {write "$(get-date -format "dd.MM.yy.HH.mm.ss") $cn.name " | out-file $LOG -append $path=get-date -format "dd.MM.yy.HH.mm" // New-Item -Path \\Hyper-V REZERV\$blade\$path -ItemType "directory" write "$(get-date -format "dd.MM.yy.HH.mm.ss") ___________________" | out-file $LOG -append Get-date | out-file $LOG -append try { $vmstate= get-vm $VmName if ($vmstate.state -eq "running"){ write "$(get-date -format "dd.MM.yy.HH.mm.ss") $VmName" | out-file $LOG —append Stop-VM $VmName -ErrorAction Stop} write "$(get-date -format "dd.MM.yy.HH.mm.ss") $VmName " | out-file $LOG -append write "$(get-date -format "dd.MM.yy.HH.mm.ss") $VmName" | out-file $LOG -append Export-VM -Name $VmName -Path \\Hyper-V REZERV\$blade\$path -ErrorAction Stop write "$(get-date -format "dd.MM.yy.HH.mm.ss") $VmName" | out-file $LOG -append Start-VM $VmName -ErrorAction Stop write "$(get-date -format "dd.MM.yy.HH.mm.ss") !!!" | out-file $LOG -append } catch {$path | Out-File $LOG —append "$_" | Out-File $LOG -append Start-VM $VmName</br> write "$(get-date -format "dd.MM.yy.HH.mm.ss") $VmName !!!" | out-file $LOG -append </br> } } else { $s = New-PSSession -computerName $i -authentication CredSSP -credential $cred Invoke-Command -Session $s -Scriptblock { $blade= hostname $LOG="C:\ClusterStorage\Volume1\ps\log\log.txt" $path=get-date -format "dd.MM.yy.HH.mm" New-Item -Path \\Hyper-V REZERV\$blade\$path -ItemType "directory" $vm="Stream.local.ru (2012r2)" write "$(get-date -format "dd.MM.yy.HH.mm.ss") ___________________" | out-file $LOG -append try { $vmstate= get-vm $vm if ($vmstate.state -eq "running"){ write "$(get-date -format "dd.MM.yy.HH.mm.ss") $vm" | out-file $LOG -append Stop-VM $vm -ErrorAction Stop} else {write "$(get-date -format "dd.MM.yy.HH.mm.ss") $vm " | out-file $LOG -append} write "$(get-date -format "dd.MM.yy.HH.mm.ss") $vm" | out-file $LOG -append Export-VM -Name $vm -Path \\Hyper-V REZERV\$blade\$path -ErrorAction Stop write "$(get-date -format "dd.MM.yy.HH.mm.ss") $vm" | out-file $LOG -append Start-VM $vm -ErrorAction Stop write "$(get-date -format "dd.MM.yy.HH.mm.ss") $vm !!!" | out-file $LOG —append } catch {$path | Out-File $LOG —append "$_" | Out-File $LOG -append Start-VM $vm write "$(get-date -format "dd.MM.yy.HH.mm.ss") $vm !!!" | out-file $LOG -append } } Remove-PSSession $s} }} } The script itself is not complicated, it has a functional for tracking errors - try catch, there are a number of messages that are written to the log and make it clear how successful the export was. There is one snag - this script runs on behalf of the domain administrator, you must tick: "for users logged in" and "run with the highest rights." Without these checkboxes, a remote connection to another host will fail. You also need to configure Powershell to export a virtual machine from one remote computer to another orb - multi-hop authentication. set up for this article . In order for hyper-v to be able to export to a network folder, it is necessary to add the name of the host from which it is being exported and give full access in its security settings. On the backup server Hyper-V REZERV, a script is running that deletes files and folders that have been modified more than 63 days. It also has Veeam Backup Free Edition, which once a month records all the folders with virtual machines on the fiber channel on the fiber channel that they exported within a month.
To summarize: a script has been created for the complete export of a virtual machine - on the one hand, this is not convenient, on the other, it gives a chance to instantly restore full server functionality. A full copy is really long, but since the hosts are separated into a separate network that is small and physically separated from the enterprise network, this does not degrade the performance of the enterprise. Some machines need to be turned off before exporting, for example, such as a database server to get a valid backup, so I advise you to make such servers small, but if this does not work, then only backup databases, for example, using MS SQL using a schedule. For smaller virtual machines, it is better to use dynamic disks - so export will take a little time.
For example, all our domain controllers use dynamic disks, and therefore such virtual machines weigh up to 20 GB - export takes 30-40 minutes. Also, some machines can be backed up “on hot”, without putting the machine into a saved state or without completing the work, but I did not find detailed articles where such export tests were described, if you know more, therefore, please write in the comments. Also, all I need to do is to check the logs - whether the script has worked or not, so I believe that such a system can really provide the company with protection against data loss and save money.
PS: I am very grateful to the users of the Technet forum, without their help, building my first cluster and setting up backups would be much more difficult. I hope that this article so far an inexperienced admin will be useful and interesting!
Source: https://habr.com/ru/post/267045/
All Articles