Half the fire did not happen: how we moved to a new data center

Two moves are equivalent to one fire.
(Popular wisdom)

Instead of the preface

A well-known cardiac surgeon arrives at the car service and rents his car for repair. The mechanic working in the workshop took this opportunity to call the doctor and ask him a question:

- Doctor! In fact, we are doing the same thing: I take out the "hearts" of cars, pull the valves out of them, put new ones. And I can replace the whole engine. Anyway, after my work, the car continues to live with a new “heart”. But you row the money with a shovel, and I get a penny for my work. Why is that?!
')
What the doctor reasonably remarked:

- And you, dear, try to make a major overhaul of a working engine!

We are growing rapidly, and we constantly need new capacities to house our equipment. At the same time, the growth of our volumes in no case should result in a decrease in the quality of our services. This is a strategic task.

Summer is the time of vacations, the most “calm” period for most webmasters: an extraordinary planned “reset” of the server is perceived more calmly.

We waited for the summer - and moved to our new data center !

Task

It would seem, what can I tell? After all, at first glance, there is nothing tricky in moving: having certain accuracy, you can easily transport anything and anywhere - especially when it comes to transporting boxes of iron, which by their nature are server and network equipment. In fact, the task of transporting hosting to a new technical platform, and even in St. Petersburg (this is an important point!), Has spicy features - in particular, it is highly desirable for hosting to continue to work during the move. Thus, the main problem that had to be solved in the process of moving was minimization of downtime in the provision of services . Based on this goal, the means were chosen.

Relocation planning was carried out based on the following data:

• Place actions - St. Petersburg. The old and new data centers are located on different banks of the Neva River at a distance of 12 kilometers from each other. For reference: in the summer at night, bridges in the city are divorced.
• It is required to transfer from the old data center to new servers and other equipment, through which virtual hosting services (shared, premium) are provided for several tens of thousands of sites, VDS rental services, dedicated servers of dedicated customers.
• The completion of the operation of the old technical site and the commencement of full operation at the new site was planned for three weeks.

Solutions

The task before us could be solved in various ways, each of which was carefully analyzed. Three main groups of solutions were identified.

Simple, cheap, clumsy

The simplest solution would look like this:

• Remove all hardware.
• To load it in the truck, to bring on a new technical platform.
• Mount it there, run it and see what happens.
• While the servers are going from place to place, make a change to the announcement of networks and continue the work of all sites at the new place using the old IP addresses.

Pros:

• It would be very cheap.
• The solution is relatively simple to implement.
• It would turn out very quickly in terms of the duration of the process of the entire move.
• There would be no need to make changes to the zones hosted on the hosting domain.

Minuses:

• The enormous risk of damage to equipment (including, all at once, as well as without the possibility of recovery) during transport.
• Significant downtime: dismantling and installation of all equipment would take several hours, which is absolutely unacceptable.

The quantitative predominance of the advantages of such a scenario of relocation could not outweigh the materiality of its disadvantages, and did not accept the option.

Laborious, expensive, elegant

An elegant solution would be to deploy a completely new technical platform in the new location - new equipment in the amount available in the old data center, new networks of IP addresses. After the readiness of the new site, it would be possible to:

• Manually transfer each site from the old server to the corresponding new one (copy files, database, mail).
• By reducing the TTL for the domain zones in advance, change the values ​​of the corresponding records.
• At the time of the DNS update, organize proxying, so that for visitors who, for one reason or another, have DNS information cached, sites will be opened from old addresses.

Pros:

• Low downtime in the work sites. For many, the move would have occurred completely unnoticed.

Minuses:

• To transfer several tens of thousands of sites, requires an indefinitely large amount of time work of qualified professionals. Given the need to do ongoing work, the move would have lasted for months.
• Not all domains of sites hosted by us are delegated to our NS servers. For those who independently maintain the zones of their domains, no elegance in this solution would be revealed, strictly the opposite.
• The time for updating the DNS information cannot be predicted or somehow influenced by it - this process does not depend on the hosting provider (for more, see the article All about domain registration and transfer ).
• Not only sites in the usual sense of the word are hosted on our hosting: a number of clients use our computing power to host specialized software, and the described approach for transferring such services would simply not be applicable.
• It would take an impressive investment in the purchase of an excess amount of new equipment, time to configure it, the cost of its maintenance.

The number of disadvantages seriously outweighs the advantages, and this decision was also considered not appropriate: no one has the opportunity to use uncontrolled processes as a tool for solving their problems.

Life

After analyzing the factors that determine the duration of a break in the provision of services, we have developed a solution that we are proud of in our depths.

Technical factors

• Our company has the status of a local Internet registry ( LIR ) and exploits its own address space. In this regard, we do not depend on Internet providers, which significantly helped us when moving. In order to avoid the need to make changes to the records of the domain zones of the sites we host, we decided to continue to work in the current address space. In order to ensure the possibility of its simultaneous use at both technical sites, the data centers were connected by a virtual network ( VLAN ). In practical terms, this gave us the opportunity to turn off the server on the old technical platform and turn it on on the new one without changing the IP addresses and without having to make changes to the routing.
• Before moving the server with the company's own services (billing, the main NS server, etc.), the operation of the backup NS server, which is located physically outside the main technical site, was additionally verified.

Organizational factors

• Transportation of hard drives was carried out separately from the servers themselves, in the new data center they were installed in new servers: this saved time for dismantling and installing equipment and SCS; In addition, it is much easier to transport hard disks at night, which is invisible to an external observer, than to transport multiple servers in the collection.
• In order to minimize the likelihood of being stopped by traffic police officers, the transportation of disks was carried out in an ordinary passenger car that was driving without any traffic violations (speed limit, etc.).
• Shared and premium servers were shipped on weekends at night - right after the bridges were closed. The time of transportation of dedicated servers was previously agreed with the clients.

Upon completion of the physical transfer of equipment to the new data center, we had only to “extinguish” the network in the old data center and make changes to the routing of our networks, which was successfully done. From the break in the work site could not be noticed. For those who, for technical reasons, noticed him, the site’s visibility “disappeared” for no more than 10 minutes.

Of the minuses of the decision adopted and enforced in life, only significant labor intensity and some overhead costs should be noted (for example, for the purchase of “buffer” equipment for a new technical platform). But these moments did not affect the qualitative side of the process, therefore, they turned out to be acceptable.

Organizational Conclusions

Of course, we didn’t succeed in “overhauling a working engine” - for objective reasons, it’s impossible to change the physical position of the equipment without interrupting its operation. But we are glad that we managed to prevent the occurrence of “half the fire” - the physical relocation of equipment by the user of the virtual hosting and most of the VDS or dedicated rental services clients looked completely indistinguishable from the ordinary full server reboot, performed, for example, to update hardware or system software : Instead of the planned two hours of downtime, which we warned customers in the mailing lists, the average time of site unavailability was 1 hour and 20 minutes.