VMWARE vSphere 5.0: Implementation Experience on the Example of the Tomsk Municipality
I am not an expert, but simply administer a small network, like any person I make mistakes, therefore I describe everything not as it should have been, but how it actually was. So some “bloopers” I have made during the time of the work done, please do not be considered too strictly.
Introduction
In the yard in 2012. I have been working for three years in a budget organization called the Tomsk City Municipality. If the first six months had to be understood and “enter” into the network organization structure and the “what” and “from” grow in terms of network equipment, then the last 2.5 years had to be spent on supporting the server “farm”. And it turned out to be quite a bit: a forest consisting of 25 domains, for every 1-2 controllers + 1-2 file servers + server with database + router if the unit is connected through a third-party provider (often the first three were performed by one server, but more on that later). And so it happened that the equipment has the habit of aging. In this case, the last time the purchase of equipment was on 01/12/2007 - two “new” servers have already noticeably aged (5 years have passed). And the allocation of new funds was enough only to support the old servers, namely, a zip-pack of hard drives in case one in the raid fails. To my constant question: “What are we going to do if the raid board itself is covered?” Experienced administrators threw up their hands. However, as time went on, there were more and more emergency situations, it also happened that during the night two hard drives in one raid took off. We (the staff of the Committee for Informatization of the Tomsk City Administration) understood that something needs to be changed. Or old servers for new ones or look towards virtualization.
')
VMWARE. Acquaintance
My acquaintance with vmware happened for the first time in my 3rd year (2004) when I went to work at the local TOMLINE office (Zmodem, evg33 - thank you) it was there that for the first time in my life I installed vmware workstation to play around with linux and freebsd and nothing do not break on working servers. Then, terribly, there was not enough processor or memory - and it looked very dull and not promising. Making an analogy, I can say that approximately the same way I met windows 3.11 when in 5th grade I ran it from Norton Commander, I looked at the “windows”, I started paint in them, and then I closed it and continued to use it as the main OS - DOS, we all know where today is DOS, and where is windows =). However, as time went on, the yard was far away in 2009, conferences and webinars on virtualization began. I put the first experiences in the work of the existing network on the vmware Infrastructure - free software from vmware, which is placed on top of windows in which you can deploy virtual machines and it “knows VLANs”. Further more, a large role was played by the free webinar from Maxim Moshkov (I still can not understand lib.ru - his project?). For the first time in an accessible language, they told, and most importantly, showed what HA, Failover, DRS, etc. are. and live demonstrated how it works and answered all my questions (very often you have to attend seminars where sales managers talk a lot on slides, when you ask a little to the side - they can’t answer anything if representatives of vmware read this: please , highlight those specialist for such presentations).
VMWARE: Implementation
The end of 2009 turned out to be quite interesting. As a result of the new mayor taking office, the entire apparatus of the city hall (most of it) switched to mobile devices. If before all that was needed was a laptop with internet and the purchase of wi-fi points on each floor completely solved all the problems and needs, then with the advent of the new leadership, the requirements grew "a little bit": the electronic workflow should work on all mobile platforms: windows mobile ( communicators), iOs (iPhones were already there and an ipad will appear a little later), symbian (still NOKIA lovers) - and all this should work and support corporate email and of course “we should be yesterday”. That's when I first encountered a shortage of servers. Service is needed. There is no server. We went to the server - to think. I see a domain controller - everything is as it should be on a separate piece of hardware, next to the server under oracle - it collects statistics on visits to the municipal site. Well, I got the idea to virtualize these two servers. And here is a freshly installed esxi 4.0 registered on the vmware site in our network. As known, a single-walled hypervisor after registration is free and this is a good marketing move. Thus, the first vmware-based hypervisor appeared in the Tomsk city administration.
Choose VMWARE
If today for me the choice of brand for virtualization is obvious, then in 2010 there was a very big debate: vmware, xen, hyper-v. Especially large this debate turned around, when for the first time questions began to arise on the acquisition of a site for virtualization, and most importantly on the allocation of funds to this site. Somehow, in a "magical" way, large server hardware suppliers from neighboring cities found out about this and began to offer and promote their solutions. Here it is worth noting the persistence of microsoft: they are great, in a distant Siberian city they were told more and more about them. Many people advertised hyper-v, but at the same time they were constantly heard: we were about to catch up with vmware - much was heard about vmware too, but mostly from webinars. But what concerns XEN is a failure, very little information (at that time). I honestly set up hyper-v, but as it turned out in 2010 it worked poorly with freebsd, namely, this operating system was used on servers located in the DMZ of our network, which was also planned to be virtualized (at the same time attending the seminar “clouds without blah blah blah "From microsoft on April 11, 2012, microsoft also showed a slight lag behind their colleagues from their vmware, the seminar began with the words:" Well, we caught up with vsphere 5.0 in our new windows server 8 ", then there are digits and digits, but it turns out that the release will be either in October, or in November, and it’s loaded from the repository I don't know how to - honestly, I don't understand why? I can subjectively look at things, but when you buy a storage that is usually more expensive than any server and you certainly have support and spare hard drives for it, why use local hard drives on hosts? For what? It is not clear. There were also attempts to study KVM, but the lack of a client under windows was the first inconvenience that was not immediately liked. So - VMWARE: everything would be fine, but here's the price ... With the most modest calculations, the license was obtained in the region of 500,000, but there would be no point in it if there was no storage and two identical server hosts for organizing a failover cluster.
We introduce a little bit
So my job duties were that first of all I had to be responsible for the operability of the forest of domains in Active Directory. When I came to work, I found about 8 “forgotten” controllers and 2 “unnecessary” domains. And if in the head domain (if you can say so) everything seemed to be fine, then errors in the directory service log fell daily. After investigating the causes of incorrectly deleted domain controllers, it became clear: problems start when tasks that they should not perform are broadcast to the domain controller. It is logical to post a domain controller, backup controller, file server, antivirus, etc. on different servers, but how to do it if there is only one physical server? This is where vmware comes to the rescue with its esxi hypervisor: on one physical server we run as many operating systems as necessary, but still depend on hardware. If something happens to the raid array, memory, motherboard, then there will be some downtime, and since the domain controller will not be available, there will be certain difficulties. There was a question about creating a dedicated server for backup controllers based on esxi.
This is the way most corporate networks (the network of the Tomsk city administration are not an exception), that each unit comes in its own optics and lives in its own subnet (in its own VLAN). If you organize an esxi server with virtual machines in the head structure and roll out the existing VLANs there, you will be able to create a backup domain controller, which some “local” administrators will not even know who will be located on the same subnets as the main controllers . In this case, if something happens to the main controller (as practice shows, it is stable once a month in one domain and it happens), then users log in to the second, there is always the possibility to deploy DHTsP, but if absolutely everything is bad with the main (burned the motherboard was what we had), we captured the roles and cleaned up the data about the old server with ntdsutil`. In any case, it is easier than creating a new domain from scratch and going to add computers to a new domain, etc. (and this we had). Actually, deploying a separate esxi host and raising such “backup controllers” on it turned out pretty quickly. He has been working steadily for a year now and everyone is used to it.
A standalone host is great. But I wanted to see all the charms of High-Availability. And if the sphere (vmware vsphere) could be used for 60 days, then where to get the storage? Here freenas and iscsi come to the rescue. In principle, a good solution, but it did not work out to achieve stability and high performance from it. I will not go into details, the article is not about that. The problems began in November 2011. For every weekend (backups are being made, disk activity is increasing), raid-arrays started to work “incorrectly”: either the hard drive will go offline, then it will be loaded only 3 times. One of the hard drives was problematic, but as long as it was calculated, the nerves were not spoiled a lot. And when this happens on the main file server, this is especially sad. And then there's the “crisis of the place”: you need to back up 500 GB, but you have nowhere to go. So you have to connect an external IDE hard drive to the server, but due to the lack of IDE power supply, you had to hold it with the lid open and power it from a regular computer. What gave the overheating of hard drives that were not covered by the fans. I had to remove the plate closing the expansion slot, and stretch the IDE cable through the resulting window. The lid was closed, the IDE hard drive on top, the temperature of the hard drives in the server dropped, the glitches stopped. And all this in anticipation of normal storage.
In the summer of 2011: as part of the target program, it is possible to upgrade the server hardware. We held two auctions and purchased one powerful server and storage. And they laid into the budget of 2012 that in January there will be another auction for the second such server, and in the second quarter we buy, again through the auction, software for virtualization. By the way, thanks to the auction, they saved 9.5% of the initial price. As a result, everything was successfully received and launched in March 2012.
Iron
As a result, we received:
Server
Intel 1U SR1695WBA platform (LGA1366, i5500, PCI-E, SVGA, SATA RAID, 4xHotSwapSAS / SATA, 2xGbLAN, 8DDR-III, 2 * 450W); Intel Xeon X5675; 64 GB RAM - 2 pcs
Controllers:
LSI Logic Host Bus Adapter SAS 9205-8e SGL (LSI00285) - 2 pcs
Qlogic iSCSI Dual Prt PCIe HBA - 3pcs
Storage System
LSI Server ACC DRIVE CHASSIS / CDE 2600-12 with integrated iSCSI and Raid controller LSI SERVER ACC Raid ISCSI SAS CRU / 2600 2 Gb M102248 - 1 pc
SERVER ACC HDD 2TB 7.2k SAS / 44354-00 M102349 PACL LSI - 12pcs
The idea: to transfer all high-loaded servers to a failover cluster consisting of two hosts (two new servers) that will be connected to the storage via SAS, the released servers (newer ones that have a PCIe slot) connected to the storage via iscsi (4 more servers). As you can see, there was not enough money for optics, and SAS seems to be fine with it. At the same time, LSI announced the release of the SAS HBA switch, which will allow connecting even more servers to the SAS storage.
Out of 3 boards ordered by iscsi, only one turned out to be fully working !!! One did not see the repository, the second, as it turned out, does not allow loading from the repository.
HBA SAS came a month later. They earned without any problems. At the moment, while the three hosts are loaded from storage. We wait while will replace under warranty QLogic-and
findings
As a result of the implementation of VMWARE vSphere 5.0 in the corporate network of the municipality of Tomsk, the following conclusions can be drawn:
1. Increased fault tolerance of critical servers by creating a High-Availabilty cluster.
2. The possibility of live migration allows you to carry out maintenance work without stopping the work of critical servers.
3. Reduction of power consumption from 5.3 kW / h to 3.8 kW / h (saving 60,000 rubles per year).
4. Ease of administration: a single management console, there is no “binding” to the hardware.
In general, if you have 3 or more stand-alone servers, then virtualization is definitely worth it.
VMWARE: PSOD comes to us
Everything is up and running. As a result of the migration, 6 old servers were turned off. The plans are still 5. Convenience of migrating virtual computers, which already 34 can not but rejoice. In general, virtualization is very relaxing - but it certainly simplifies the administrator’s life. And I came on Tuesday April 24, 2012. As usual I come to work and I see an alarm from Nagios that one of the hosts is not available. The monitor also showed me this picture:
A little bit of Google and everything is in principle clear. Communities.vmware.com/thread/311457?start=0&tstart=0 (in general, the vmware knowledge base is good, but you can’t tell about their download center). But we have the same mandatory support. Under the terms of VMware, support is provided only after registering the product: ok, register. We get another interesting screenshot.
We contact suppliers, we are recommended to unsubscribe by email: vi-hotline@vmware.com
We are writing a letter. We get the answer.
Thank you for requesting support from VMware Licensing Support. Your Support Request Number is 12167833904
Your Licensing Support Specialist will respond to your query within 12 hours.
Now 12 hours have passed, but there was no answer. It's a pity. Well, that's basically all. It is morning April 25th. And this modest essay on the experience of implementing VMWARE vSphere 5.0 on the example of the Tomsk municipality came to an end. If you have any questions, suggestions, or a desire to share experience, write gtt@admin.tomsk.ru I will try to answer everyone. Thank you for reading this article to the end.
Introduction
In the yard in 2012. I have been working for three years in a budget organization called the Tomsk City Municipality. If the first six months had to be understood and “enter” into the network organization structure and the “what” and “from” grow in terms of network equipment, then the last 2.5 years had to be spent on supporting the server “farm”. And it turned out to be quite a bit: a forest consisting of 25 domains, for every 1-2 controllers + 1-2 file servers + server with database + router if the unit is connected through a third-party provider (often the first three were performed by one server, but more on that later). And so it happened that the equipment has the habit of aging. In this case, the last time the purchase of equipment was on 01/12/2007 - two “new” servers have already noticeably aged (5 years have passed). And the allocation of new funds was enough only to support the old servers, namely, a zip-pack of hard drives in case one in the raid fails. To my constant question: “What are we going to do if the raid board itself is covered?” Experienced administrators threw up their hands. However, as time went on, there were more and more emergency situations, it also happened that during the night two hard drives in one raid took off. We (the staff of the Committee for Informatization of the Tomsk City Administration) understood that something needs to be changed. Or old servers for new ones or look towards virtualization.
')
VMWARE. Acquaintance
My acquaintance with vmware happened for the first time in my 3rd year (2004) when I went to work at the local TOMLINE office (Zmodem, evg33 - thank you) it was there that for the first time in my life I installed vmware workstation to play around with linux and freebsd and nothing do not break on working servers. Then, terribly, there was not enough processor or memory - and it looked very dull and not promising. Making an analogy, I can say that approximately the same way I met windows 3.11 when in 5th grade I ran it from Norton Commander, I looked at the “windows”, I started paint in them, and then I closed it and continued to use it as the main OS - DOS, we all know where today is DOS, and where is windows =). However, as time went on, the yard was far away in 2009, conferences and webinars on virtualization began. I put the first experiences in the work of the existing network on the vmware Infrastructure - free software from vmware, which is placed on top of windows in which you can deploy virtual machines and it “knows VLANs”. Further more, a large role was played by the free webinar from Maxim Moshkov (I still can not understand lib.ru - his project?). For the first time in an accessible language, they told, and most importantly, showed what HA, Failover, DRS, etc. are. and live demonstrated how it works and answered all my questions (very often you have to attend seminars where sales managers talk a lot on slides, when you ask a little to the side - they can’t answer anything if representatives of vmware read this: please , highlight those specialist for such presentations).
VMWARE: Implementation
The end of 2009 turned out to be quite interesting. As a result of the new mayor taking office, the entire apparatus of the city hall (most of it) switched to mobile devices. If before all that was needed was a laptop with internet and the purchase of wi-fi points on each floor completely solved all the problems and needs, then with the advent of the new leadership, the requirements grew "a little bit": the electronic workflow should work on all mobile platforms: windows mobile ( communicators), iOs (iPhones were already there and an ipad will appear a little later), symbian (still NOKIA lovers) - and all this should work and support corporate email and of course “we should be yesterday”. That's when I first encountered a shortage of servers. Service is needed. There is no server. We went to the server - to think. I see a domain controller - everything is as it should be on a separate piece of hardware, next to the server under oracle - it collects statistics on visits to the municipal site. Well, I got the idea to virtualize these two servers. And here is a freshly installed esxi 4.0 registered on the vmware site in our network. As known, a single-walled hypervisor after registration is free and this is a good marketing move. Thus, the first vmware-based hypervisor appeared in the Tomsk city administration.
Choose VMWARE
If today for me the choice of brand for virtualization is obvious, then in 2010 there was a very big debate: vmware, xen, hyper-v. Especially large this debate turned around, when for the first time questions began to arise on the acquisition of a site for virtualization, and most importantly on the allocation of funds to this site. Somehow, in a "magical" way, large server hardware suppliers from neighboring cities found out about this and began to offer and promote their solutions. Here it is worth noting the persistence of microsoft: they are great, in a distant Siberian city they were told more and more about them. Many people advertised hyper-v, but at the same time they were constantly heard: we were about to catch up with vmware - much was heard about vmware too, but mostly from webinars. But what concerns XEN is a failure, very little information (at that time). I honestly set up hyper-v, but as it turned out in 2010 it worked poorly with freebsd, namely, this operating system was used on servers located in the DMZ of our network, which was also planned to be virtualized (at the same time attending the seminar “clouds without blah blah blah "From microsoft on April 11, 2012, microsoft also showed a slight lag behind their colleagues from their vmware, the seminar began with the words:" Well, we caught up with vsphere 5.0 in our new windows server 8 ", then there are digits and digits, but it turns out that the release will be either in October, or in November, and it’s loaded from the repository I don't know how to - honestly, I don't understand why? I can subjectively look at things, but when you buy a storage that is usually more expensive than any server and you certainly have support and spare hard drives for it, why use local hard drives on hosts? For what? It is not clear. There were also attempts to study KVM, but the lack of a client under windows was the first inconvenience that was not immediately liked. So - VMWARE: everything would be fine, but here's the price ... With the most modest calculations, the license was obtained in the region of 500,000, but there would be no point in it if there was no storage and two identical server hosts for organizing a failover cluster.
We introduce a little bit
So my job duties were that first of all I had to be responsible for the operability of the forest of domains in Active Directory. When I came to work, I found about 8 “forgotten” controllers and 2 “unnecessary” domains. And if in the head domain (if you can say so) everything seemed to be fine, then errors in the directory service log fell daily. After investigating the causes of incorrectly deleted domain controllers, it became clear: problems start when tasks that they should not perform are broadcast to the domain controller. It is logical to post a domain controller, backup controller, file server, antivirus, etc. on different servers, but how to do it if there is only one physical server? This is where vmware comes to the rescue with its esxi hypervisor: on one physical server we run as many operating systems as necessary, but still depend on hardware. If something happens to the raid array, memory, motherboard, then there will be some downtime, and since the domain controller will not be available, there will be certain difficulties. There was a question about creating a dedicated server for backup controllers based on esxi.
This is the way most corporate networks (the network of the Tomsk city administration are not an exception), that each unit comes in its own optics and lives in its own subnet (in its own VLAN). If you organize an esxi server with virtual machines in the head structure and roll out the existing VLANs there, you will be able to create a backup domain controller, which some “local” administrators will not even know who will be located on the same subnets as the main controllers . In this case, if something happens to the main controller (as practice shows, it is stable once a month in one domain and it happens), then users log in to the second, there is always the possibility to deploy DHTsP, but if absolutely everything is bad with the main (burned the motherboard was what we had), we captured the roles and cleaned up the data about the old server with ntdsutil`. In any case, it is easier than creating a new domain from scratch and going to add computers to a new domain, etc. (and this we had). Actually, deploying a separate esxi host and raising such “backup controllers” on it turned out pretty quickly. He has been working steadily for a year now and everyone is used to it.
A standalone host is great. But I wanted to see all the charms of High-Availability. And if the sphere (vmware vsphere) could be used for 60 days, then where to get the storage? Here freenas and iscsi come to the rescue. In principle, a good solution, but it did not work out to achieve stability and high performance from it. I will not go into details, the article is not about that. The problems began in November 2011. For every weekend (backups are being made, disk activity is increasing), raid-arrays started to work “incorrectly”: either the hard drive will go offline, then it will be loaded only 3 times. One of the hard drives was problematic, but as long as it was calculated, the nerves were not spoiled a lot. And when this happens on the main file server, this is especially sad. And then there's the “crisis of the place”: you need to back up 500 GB, but you have nowhere to go. So you have to connect an external IDE hard drive to the server, but due to the lack of IDE power supply, you had to hold it with the lid open and power it from a regular computer. What gave the overheating of hard drives that were not covered by the fans. I had to remove the plate closing the expansion slot, and stretch the IDE cable through the resulting window. The lid was closed, the IDE hard drive on top, the temperature of the hard drives in the server dropped, the glitches stopped. And all this in anticipation of normal storage.
In the summer of 2011: as part of the target program, it is possible to upgrade the server hardware. We held two auctions and purchased one powerful server and storage. And they laid into the budget of 2012 that in January there will be another auction for the second such server, and in the second quarter we buy, again through the auction, software for virtualization. By the way, thanks to the auction, they saved 9.5% of the initial price. As a result, everything was successfully received and launched in March 2012.
Iron
As a result, we received:
Server
Intel 1U SR1695WBA platform (LGA1366, i5500, PCI-E, SVGA, SATA RAID, 4xHotSwapSAS / SATA, 2xGbLAN, 8DDR-III, 2 * 450W); Intel Xeon X5675; 64 GB RAM - 2 pcs
Controllers:
LSI Logic Host Bus Adapter SAS 9205-8e SGL (LSI00285) - 2 pcs
Qlogic iSCSI Dual Prt PCIe HBA - 3pcs
Storage System
LSI Server ACC DRIVE CHASSIS / CDE 2600-12 with integrated iSCSI and Raid controller LSI SERVER ACC Raid ISCSI SAS CRU / 2600 2 Gb M102248 - 1 pc
SERVER ACC HDD 2TB 7.2k SAS / 44354-00 M102349 PACL LSI - 12pcs
The idea: to transfer all high-loaded servers to a failover cluster consisting of two hosts (two new servers) that will be connected to the storage via SAS, the released servers (newer ones that have a PCIe slot) connected to the storage via iscsi (4 more servers). As you can see, there was not enough money for optics, and SAS seems to be fine with it. At the same time, LSI announced the release of the SAS HBA switch, which will allow connecting even more servers to the SAS storage.
Out of 3 boards ordered by iscsi, only one turned out to be fully working !!! One did not see the repository, the second, as it turned out, does not allow loading from the repository.
HBA SAS came a month later. They earned without any problems. At the moment, while the three hosts are loaded from storage. We wait while will replace under warranty QLogic-and
findings
As a result of the implementation of VMWARE vSphere 5.0 in the corporate network of the municipality of Tomsk, the following conclusions can be drawn:
1. Increased fault tolerance of critical servers by creating a High-Availabilty cluster.
2. The possibility of live migration allows you to carry out maintenance work without stopping the work of critical servers.
3. Reduction of power consumption from 5.3 kW / h to 3.8 kW / h (saving 60,000 rubles per year).
4. Ease of administration: a single management console, there is no “binding” to the hardware.
In general, if you have 3 or more stand-alone servers, then virtualization is definitely worth it.
VMWARE: PSOD comes to us
Everything is up and running. As a result of the migration, 6 old servers were turned off. The plans are still 5. Convenience of migrating virtual computers, which already 34 can not but rejoice. In general, virtualization is very relaxing - but it certainly simplifies the administrator’s life. And I came on Tuesday April 24, 2012. As usual I come to work and I see an alarm from Nagios that one of the hosts is not available. The monitor also showed me this picture:
A little bit of Google and everything is in principle clear. Communities.vmware.com/thread/311457?start=0&tstart=0 (in general, the vmware knowledge base is good, but you can’t tell about their download center). But we have the same mandatory support. Under the terms of VMware, support is provided only after registering the product: ok, register. We get another interesting screenshot.
We contact suppliers, we are recommended to unsubscribe by email: vi-hotline@vmware.com
We are writing a letter. We get the answer.
Thank you for requesting support from VMware Licensing Support. Your Support Request Number is 12167833904
Your Licensing Support Specialist will respond to your query within 12 hours.
Now 12 hours have passed, but there was no answer. It's a pity. Well, that's basically all. It is morning April 25th. And this modest essay on the experience of implementing VMWARE vSphere 5.0 on the example of the Tomsk municipality came to an end. If you have any questions, suggestions, or a desire to share experience, write gtt@admin.tomsk.ru I will try to answer everyone. Thank you for reading this article to the end.
Source: https://habr.com/ru/post/143663/
All Articles