Personal experience: how we transferred the infrastructure from one data center to the US to another

We in King Servers are faced with the need to change one of our foreign data centers. The data center was located in the US state of California, and it was necessary to look for a new one in the same region. Today we will talk about how the relocation took place, share our impressions of it and give a couple of tips to those who still have to face the infrastructure sites in the United States.

Why change the data center

The development of the company's infrastructure has gradually led to the fact that we are no longer satisfied with the conditions of our current data center. Despite the fact that the data center had a Tier2 standard, in just a year we faced a number of problems, such as power failure in some of the racks or a complete drop in the Internet channel.
')
In addition, the site was not Carrier-neutral - in practice, this meant that communication services were provided only by one company that owned the data center. Therefore, in the moments of failures on its channels, the connection disappeared completely, which could not suit us or our clients.

After weighing all the pros and cons, we decided to migrate the infrastructure to a new data center, where it would be possible to use two independent power lines simultaneously and connect to several providers.

Process of moving

Both data centers were located close to each other, at a distance of 20 km. Old in the city of Fremont, and the new in Santa Clara.



Preparing for the move thoroughly - pre-purchased consumables required for mounting servers in racks, new power cables and patch cords to reduce the time for dismantling in the old data center.



Relocation stocks

We also installed a second router in a new data center and organized the simultaneous operation of our networks in both data centers.

As a result, a plan was developed, involving a phased transfer of infrastructure to a new location - so we could continue to work and not cause unnecessary inconvenience to users.

Initially, we wanted to use the services of a specialized company for the transportation of equipment, but in the process of negotiations with several companies we found out that they could transport our equipment either in large quantities with large downtime for customers, or small parts with minimum downtime, but the price increased significantly under such conditions. As a result, it was decided to send several of our engineers across half the globe and perform the work on their own. Even taking into account flights, accommodation and transport rent, such a move was 2 times cheaper than the minimum price list from Californian moving companies.

The first stage of work included an exit to the old data center to assess the infrastructure - not all engineers of our team have been there before, and some last time came a couple of years ago, and during this time many different works were carried out by data center employees. It was necessary to check and install the equipment - as it turned out, not everything was as good as on our last visit.

The racks used cables with excess length, some servers were mounted without using a sled and tightly screwed to the rack. It became clear that the time for dismantling would take much more than we had planned. Fortunately, the spare materials were enough to do without the complete dismantling of all the iron, and we still fit into the migration schedule.



We can not say that we were pleased with what he saw in the old DC

This was followed by a reconnaissance trip to the new data center to look around and explore the new racks.

During the first visit to the new data center, our engineers first of all evaluated its manufacturability. In the old data center everything was very strict - the data center employees not only checked the engineers' documents very meticulously, but also accompanied them when they moved to and near their racks. This entailed temporary delays, as it was often necessary to wait until such an attendant was free. The new site turned out to be much more democratic in this regard, which allowed us to work faster.

According to the plan, the team of engineers had to come to the old data center, pick up 15-20 servers, then install them in racks at the new location, turn them on, and after that everything had to work without requiring additional intervention from our side.

Not without difficulties - for example, it suddenly turned out that the Cisco router that we planned to use to organize the tunnel did not work the way it was planned after transporting the test part of the servers not used by clients and testing, it became clear that it would not sustain the traffic of the clients.

We had to quickly change the network layout and use additional routers to share the load from the clients and allow the network to last a couple of days during the migration. The setup time was no longer included in any of our plans, which nevertheless caused a deviation from the schedule.

Also, not all servers started immediately after transportation. Some of our clients administer dedicated servers independently and close all access to us, including monitoring. On some servers, the disks were not in the best condition, and even a neat shutdown with the correct shutdown did not save the system from failures due to damaged file systems and crashed raids.

In each new batch of the transported servers, a couple of similar problems were encountered, and the technical support engineers after the migration of each batch of servers carried out work to restore their functionality.

Total: general impressions

In American data centers everything is very strict, it is not easy to get even minimal access to your own equipment - for every action you need to get approval, which can only be given by a manager who is not necessarily physically located in the DC. Then you need to create a ticket in the helpdesk system and wait for its resolution.

You can also meet quite tough conditions, for example, on the use of data center resources - so in our new data center there is a restriction that prohibits keeping your hardware in a warehouse for a long time (more than 1-2 days). We needed more time, and in order to get permission for longer storage, we had to obtain manager approval by the scheme described above. ID (passport or rights), as in principle in the United States, is constantly being asked at local data centers; you also need to be prepared for this.

On the other hand, from an organizational point of view, the use of DC in California turned out to be very pleasant. All the conditions stipulated in the contracts were always fulfilled, which we did not always manage to achieve in Russia. For example, once our move to a new data center failed due to the fact that its engineers did not have time to prepare the racks - although the terms of preparation were clearly specified in the contract (we told about our adventures in this material ).

In addition, local engineers were very friendly and always met us halfway - for example, when we had a shortage of cables so that we would not be interrupted and go to the store to buy new ones, our colleagues allocated resources from their stocks to us.

Conclusion: what you need to know when working with American data centers

In conclusion, we list it again somewhat

• In the US, not all data centers are very technically advanced - so you should not choose the first available option, but rather spend more time searching. Perhaps, the cost of services of a more innovative data center will be more expensive, but at a certain size of infrastructure, better conditions may bring savings on the repair, maintenance and replacement of iron.
• It is necessary to strictly follow the rules - the United States is a country where laws are enforced, even if it seems not always logical. But if the rules say that the client cannot move two steps away from his desk, then in order to do this, the manager’s permission must be obtained — and in the process sign a number of papers and show the ID several times.
• Everyone is very friendly, but in any case, you only need to hope for yourself - the famous American positivity is not a myth, and it applies to data center engineers, who always don’t mind helping. However, this does not mean that you need to relax - for example, no one will fix for you when preparing for migration.

More useful links and materials from King Servers :

• Free infrastructure test from King Servers
• How and why we created a knowledge base
• How to choose a data center for an IT project in Russia: uptime, money and overall adequacy
• Popularity statistics of operating systems in IaaS: Ubuntu is still number one, CentOS popularity is growing
• When the “cloud” breaks down: what can be done in this situation?