How it works: monitoring of power supply of data center Selectel

What do you think that these three events can have in common? A Vkontakte user posted a new photo, someone else made a purchase online with a payment card through the Assist system, and a third person drove his car into CarPrice and its price appeared in the auction tape?

And each of these people got the result they expected?

The answer is: all these actions became possible due to the fact that here we miss a lot of intermediate levels and interconnections ... that the servers with these applications in the Selectel * data center are supplied with reliable and uninterrupted power supply, and its parameters are monitored all the time. Yes, no matter how primitive it may sound, the power industry infrastructure is at the core of the entire IT industry. We already told

about the data center power supply system in details in the article The path of electricity , and now is the time to cover the topic of monitoring.
* These brands also use the power of their own IT resources to serve their customers.

Monitoring against the 2nd law of thermodynamics

It would not be an exaggeration to say that the power supply infrastructure is just an area in which, in all its beauty or, more correctly, ugliness, the 2nd law of thermodynamics manifests itself: "Any system tends to chaos, if not to spend energy on maintaining order. " This is the most important task - to prevent chaos in the power supply of the data center - and is entrusted to numerous multi-level power parameters monitoring systems and the Selectel technical service team.

')

Let's see how the electricity from the city-wide network passes through all levels of power supply to the Selectel Floral Center 2 data center (here it is in the photo) and reaches the power supply unit of a specific server or disk storage. And, of course, let us tell you about what power supply parameters are measured at each level and stage, because it is impossible to control what is not measured.

Why did we decide to talk about power monitoring? We understand that the company's customers are not interested in abstract, albeit grandiose systems in the data center infrastructure, but the smooth operation of their equipment. In turn, for Selectel as a commercial organization, the most important criterion for business operations are satisfied customers - it is you who help us with our money to grow, and your comments and ideas to improve the quality of the company's services.

Tour of the monitoring system

Probably, it is not interesting to anyone to read in a blog dry excerpts from instructions and regulations on power supply monitoring, isn’t it? So welcome to the excursion!

The company Selectel has 6 data centers (data centers, data centers), and we will tell about the power supply using the example of the data center “Flower 2” in St. Petersburg. In the other five data centers, “Tsvetochnaya 1” in St. Petersburg, “Dubrovka 1-2-3” in the Leningrad Region and “Berzarin 1-2” in Moscow, electricity is supplied and monitored according to similar principles and regulations. .

Monitoring begins from the moment electricity is supplied to the data center. Two independent electrical cables, so-called feeders, were laid to the building of the Tsvetochnaya 2 data center. For each of the feeders, a high voltage of 6 kV (in other data centers of 10-20 kV) comes to the data centers on Flower Street. To start using it for the needs of the data center, the voltage is first lowered to 380 V at two transformer substations TP1 / TP2 of consumer class TP 6-10 / 0.38 kV, located on the territory of the data center. The voltage and frequency of the electric current are measured both at the input to the substation and at the output. The temperature inside the substations is also measured, which should not exceed certain limits.

From transformer substations, external power falls on two groups of uninterruptible power supplies (UPS), working in conjunction with blocks (matrices) of 12 V batteries. The uninterruptible power supply monitors the input voltage, UPS status, inverters, capacitors and fans. At the output of UPS clusters, we measure and monitor voltage and frequency.

Battery packs are lead 12 V batteries, akin to car batteries, and interconnected into huge dies. By the way, remember the discussion on the Internet that supposedly the batteries for Tesla cars are collected from hundreds of finger-type batteries? So, backup power for the data center is provided exactly this way - more than a hundred ordinary 12 V batteries. The temperature and voltage on the battery packs are checked in real time. If any 12 V battery fails, Selectel’s technical service will find out about it in a timely manner and replace it.

If at both TP1 / TP2 substations the input voltage supplied from the city completely disappears, the capacity of the battery packs is enough for 10 minutes of data center operation under full load. During this time, diesel generator sets (DGS) are started, which take over the entire load. In addition to the parameters of the output voltage from the DGU, many parameters are monitored on these installations. These are engine revolutions, oil temperature and pressure, coolant temperature, fuel level, active and full power output, total hours run and other parameters.

DGU installations are constantly in a preheated state, the temperature of oil and coolant in motors is maintained at + 40C. This makes it easy to start diesel engines in the cold season, and within one minute to ensure fast warm-up and access to full load.

The power supplied from the UPS is then fed into the machine rooms on the power cabinets and installed in racks for servers, storage systems and network equipment. Two independent, interchangeable 220 V power supplies come inside the racks. This is done in order to be able to install servers and storage systems with two power supplies in a rack according to the 2N redundancy scheme. Input power racks are also measured and monitored — voltage and power consumption. Thus, the Selectel technical service knows the power consumption of both the entire data center and each individual rack.

If IT equipment with only one power supply is installed in a rack (for example, the customer has such equipment), then you can optionally use a rack-level automatic transfer switch (automatic reserve transfer unit), which in case of a voltage drop on one of the inputs, switches the power to the other rack-mount entrance. This happens so fast that the work of IT equipment is not interrupted. AVR status is also monitored, and Selectel technical service has all reliable information about the power supply of each rack.

We traced the entire power path of the data center, starting from the outer urban substation and ending with a specific rack in the engine room. And now our tour will visit the management and monitoring center of the data center infrastructure, where we will also get acquainted with the staff of Selectel, who around the clock monitor the reliable power supply of equipment used by our customers.

Our heroes: engineering team

Interchangeability, support of each other and mentoring are the basic principles of the work of the engineering and technical department (ITO) team. The department employs about 40 specialists who are divided into three shifts across different data centers. If necessary, employees can come to the aid of each other, taking on the task of monitoring the parameters of power supply and ventilation in the remote mode, help each other with advice.

Next, we will see a video of how the ITO team works in real life from two different points of view. Tatiana Svirko, ITO Head, talks about the process entirely at the main console of monitoring the power supply and ventilation parameters of the data center:

And now we will bypass the critical points of the data center infrastructure and look at everything through the eyes of engineer Dmitry Kostryukov:

This is how thoroughly the entire infrastructure of the Selectel data center is checked and monitored. Our customers can be sure of their equipment!