Mass client migration between hosting providers: struggling with entropy on an industrial scale

In the life of any hosting provider, transferring clients between their own servers is a fairly common task. For such a transfer, there can be many reasons: starting with a planned upgrade of equipment or software and a planned "redistribution" of clients due to uneven server load, ending with the urgent movement of users in case of accidents.

Less often in the life of providers, there are tasks for transferring clients from another shared-hosting provider.
The reason for such a transfer can be either the “fatigue” of the hosting provider “donor” from such a high-tech business, or the outsourcing of hosting services to a larger hosting company if the provision of these services is not for the organization that has decided to change if we are talking about a web-studio, Internet service provider, service provider, etc.).
The number of potential problems when transferring clients between different providers is much greater than when transferring within a single hosting provider. This is due to the fact that the infrastructure of the “old” and “new” providers may differ significantly:

• Different billing panels are used;
• Different hosting control panels are used;
• Different versions of PHP / other interpreters with different settings;
• Different modes of running scripts (fastcgi vs mod_something);
• Different web server settings;
• A different scheme for the inclusion of web servers (with or without using frontend)
• Different settings and differences in security policy;
• Different paths in the system to the sites and external utilities;
• Different mail servers and mail sending schemes (sending sendmail to 25 port vs sendmail).

At the same time, it would not be desirable to completely copy someone else's infrastructure, which may turn out to be suboptimal or having any limitations. Even if the initial infrastructure is of high quality and interesting, there is no point in producing a “zoo” - one can only borrow interesting finds in order to implement them in the future. Thus, it is usually necessary to transfer sites to a completely new environment and map the “alien” infrastructure to “your own”.
')
There are also various potential "inconsistencies" from real life, taking into account the possibilities of small hosting providers:

• Different options and different amounts of resources on the tariff plans (1: 1 tariffs are almost never combined);
• Different clients of one hoster may live in different hosting / billing panels, and some customers may be entered “in Excel” in general, and their payment will be monitored manually (of course, some of this information may not be relevant);
• Sites that are physically on the servers, but not in the billing;
• The same clients can be duplicated on different physical servers. For example, if the client was transferred from the server to the server, and forgot to delete it from the old server;
• Different sites of the same hosting client (having a single account in the hosting panel) can be served on different servers;
• Other exotic options that need to somehow resolve.

Naturally, we have to deal with all these problems and somehow resolve them. At the same time, it is necessary to do it in such a way that none of the clients "should not be hurt": no one's websites would "fall", and the tariff conditions would not worsen. In general, so that “no one is forgotten and nothing is forgotten” and, at the same time, everything continues to work.

In fact, in this case, you need to solve three problems :

• Display foreign infrastructure on your own, so that everything continues to work;
• "Restore order", eliminating all possible inconsistencies and getting rid of technological chaos;
• Minimize downtime when migrating.

In this review article I will try to tell how these problems are solved in REG.RU.
Usually the process consists of several technological, organizational and informational stages :

1. Notification of customers about the upcoming transfer:
a) News on the site hosting;
b) News on REG.RU;
c) Press release;
d) Newsletter to customers about the upcoming transfer;
e) Pause 1-2 weeks. This is done so that those who do not want to be transferred can declare this or independently switch to another hosting provider.

The goal of the stage is to exclude all sorts of surprises so that customers understand that they will be transferred to a new provider while improving the conditions of service. In addition, users need to make it clear that the transfer will occur automatically, and every effort will be made to minimize downtime. At the same time, it is necessary to inform you that you can refuse to transfer or go to an alternative hoster with a refund for the unused paid period.

2. Technical preparation for the transfer.
2.1. Compilation of customer base and services for transfer:
a) We read and convert to the common format the data from the billing panels (of which there can be several);
b) Collect data from other sources (customers without billing);
c) Among overdue hosting sites, which are not yet physically remote, we isolate customers who actually do not use hosting anymore and whom it does not make sense to transfer, most likely they will no longer extend hosting.
It would be possible, say, to check the delegation of a domain for IP hosting, but not everything is so simple - there are situations when:

• The client forgot to renew the domain, although he plans to use the services;
• The client uses hosting only for mail (then you only need to watch MX);
• The client uses hosting to store some personal files;
• The client uses the hosting for tests or some kind of training and at the same time does not use the domain (it edits the hosts to access the site).

So, ideally, you need to look at the logs if there are any real visits to the site.

The goal of the stage is to compile an accurate list of customers and services for transfer, including domain registration services. In addition, it is necessary to collect all data for billing, including customer contact information, information on account balances, services paid for and payment periods, as well as tariffs and tariff options for services related to billing.

2.2. Entering data about customers and their services into the billing database:
a) Generation of a username (prefix with the hostname, suffix - login from the previous hoster or converted e-mail, if there is no login);
b) Password generation;
c) Transfer of contacts;
d) Transfer of balances on personal accounts / data on the dates of service disconnection;
e) Creating domains (it is possible to add API to third-party registrars if the domains are in other registrars and their mass transfer is impossible);
e) Creation of services. We take into account the display of tariffs at the previous hoster and with us. If the client does not fit in the tariff limits - we raise the tariff "at our expense").

The goal of the stage is to add information about all customers and their services to the billing database.

2.3. Preparing for physical migration / splitting clients across servers:
a) We compile a complete and “final” list of accounts that need to be migrated;
b) Putting order in the previously created list (you can say that we put things in order on the hosting, from where we transfer accounts): we look to see if there are not two or more servers of the same users or www-domains on two or more servers. If such are available - in the semi-automatic and manual mode we understand and remove the excess;
c) We divide the entire list of users into groups that satisfy the following conditions:

• There should be no more than n users in one group;
• In one group, the total limits of the resources used should not be exceeded;
• To facilitate the subsequent proxying of traffic from the old servers to the new ones (which will turn on after the transfer), it is also advisable not to split accounts from the same source server into different groups.

d) We verify the settings of the source servers and the allowed rights on the source with the settings and rights on the destination server.

The goal of the stage is to distribute clients between the destination servers, taking into account all the limitations and nuances.

2.4. Transfer accounts between hosting panels:
a) Preparation of lists of sites for transfer (we consider aliases, redirects);
b) Transferring account settings (version and PHP startup mode, etc.);
c) Transfer of mail settings (mail accounts, mail groups, redirects);
d) Transferring the DNS settings (simultaneously changing the IP addresses of the servers of the old infrastructure to new ones, leaving no changes to the additional DNS entries made by the client);
e) Renaming / ordering database names. We prepare the replacement table (old database name => new), adding a prefix with the user ID. We pay special attention to:

• Databases with names less than n characters;
• Databases with names like sql, mysql, new, user, etc., which cause confusion in the namespace of databases.

For such bases, additional control and preparation of a manual patch for replacing names from old to new ones is needed, since the automaton replacing the alternate bases in scripts and configs is highly likely to make a mistake by confusing them with other strings.

The goal of the stage is to create all the necessary accounts and their settings in the database of the “hosting” hosting panel.

3. Physical transfer of sites:
For each group of clients intended for transfer to the next server:
a) We start the automatic migration mechanism of one group;
b) Run the automatic script for automatic reconfiguration of the site configs for the new environment:

• We change the old names of the bases for the new ones (see p. 2.4 (e));
• Change old paths to files / directories for new ones;
• We resolve conflicts when the machine doubts the replacement is correct: in an amicable way, you need to drive out the machine before transferring and prepare a manual patch for questionable places that is currently rolling.

d) We launch the automatic machine for testing the main pages of the sites after the transfer: the automatic machine looks that the HTTP statuses of the main pages of all the sites have not changed;
c) Additionally we carry out a manual check - we give the list of www-domains of this group to the hosting support service / testing service;
e) Edit the found bugs;
f) Enable proxying for domains of the migrated group to us, block clients of this group at the source.

The purpose of the stage is the physical transfer of files and databases of sites, as well as the contents of mailboxes to new servers, change of script settings due to the changed environment, the physical launch of sites on the new server and the traffic going there.

4. Actions after transfer:
a) We send customers a notification with their new account details to access the billing / control panels and features of the new hosting, which should be paid attention to.
b) Final control of the transfer correctness, correction of errors;
c) Automatically change DNS servers to ours (for those domains that are under our control);
d) We distribute to clients with their own DNS with a request to change the DNS to ours (for domains outside our reach).

The purpose of the stage is to change DNS to new IP and send all necessary notifications to customers.

One of the latest examples of such a massive transfer of clients is cooperation with Hostingjoomla.ru , as a result of which more than 500 clients (750+ www-domains) were transferred from several old servers to new ones. Along the way, a lot of work was done to combat chaos (elements of this work were listed above).

As a result, everyone got the desired PROFIT:

• Web Studio RedSoft (which owns the project Hostingjoomla.ru), which specializes in launching projects based on CMS Joomla !, transferred hosting services to a specialized operator and got the opportunity to fully focus on the core business, relying on a more stable platform for their clients' projects;
• Clients received a better and more stable hosting service at a favorable price, round-the-clock quality technical support and a much larger number of options and additional services (dedicated IP addresses, SSL certificates, domains in more than 200 zones, etc.) that can be order in the format of "one window" (one account in REG.RU);
• REG.RU has received new loyal customers aimed at long-term cooperation, because web sites rarely go off from shipyards of web studios;
• REG.RU engineers received additional experience points and the next addition to a set of automatic scripts to resolve difficult situations associated with the transfer.

But the story with Hostingjoomla.ru is not over - we optimized for the needs of CMS Joomla! line of existing tariffs , added the ability to order Joomla-tariffs in our API, finalized REG.Panel , also adding new tariffs there and even choosing the Joomla version for auto-installation. As a result, web studio clients received a product that was fully optimized for their needs. And this is PROFIT for the entire market.

In general, the transfer of hosting infrastructure from small providers to larger ones has become a prevailing trend in the Russian market, as this is a chance to transfer the hosting infrastructure to reliable hands, having received in return the opportunity to focus on other tasks. Any questions - write: we will help, we will help!