Overview of IXcellerate data center (the largest machine building in the Russian Federation)

The most important thing is that after the main hall, hot wet wipes are served here, as in Japanese restaurants. This has nothing to do with the technical part, but is humanly enjoyable.

And this is a guitar with an autograph by Paul McCartney, and yes, they hang it at the entrance to the gym:
')


This data center has a certain mood.

It looks like a huge hangar in the industrial zone near the metro station "Otradnoe". The hangar height is 14 meters, inside it is built a data center with a height of 9 meters. The remaining space plays the role of a heat insulator, which affects local cooling features. The building was chosen so that it did not enter the road by edges (it’s impossible to get high levels according to certification protocols, if you can get to the machine room by truck, ramming the wall, there were probably cases), had two different check-in routes and were relatively close to the metro station. In the same place, the ICC station was opened nearby, but the road leads to such fun places ...



Canary. We like everything here so far, so the story is generally positive. But I vow to promise that if something goes wrong, we will be the first to tell about it. Well, or simply remove this paragraph from the post, if we are bound by the NDA.

What is IXcellerate data center

We have a cloud of Technoserv. This is a natural cloud for Russian customers, where there are segments certified by PCI DSS and in the class for “heavy” PD and state information systems. There are also commercial customers in the cloud who need S3 and AWS API in Russia. Until this year, we provided a service from the DataPRO data center, about which there was already a short story .

Naturally, no matter how cool the data center is, you always need a second one in case the meteorite hits the first one. Geo-distributed area - a guarantee of health and long life.

We chose the second site as carefully as the first one. We needed T3 UI (more precisely, regulations from it), the ability to work with PCI DSS (that is, our own certificate for the physical protection of the data center), and a host of other features. As a result, we got up at the IXcellerate data center - this is where the Orange stands are bricked up inside one of the machine halls, there is a peering * one very large gaming company *, there are large banks and state-owned companies, Reuters and Nestle. "Kommersant" wrote that Apple and Booking for Russian personalities have also chosen this place. In general, if they have something to fall, they just have a lid. That is, the motivation to do everything correctly is good enough.

The second feature - the head is not new. Far from new. Guy Wilner is an entrepreneur from England. He builds data centers is not the first and not the tenth time. Prior to the project, he owned his own network of 14 data centers in Western Europe, and then sold it to Equinix (data center number 1 operator in the world). After that, he decided to repeat the adventure and open a huge data center in Brazil, Russia or Turkey. He was not allowed to go to Brazil by his wife and children (because he was needed at home), and in Russia he met Cliff Gauntlett. This is a legendary personality in RuNet in the field of telecommunications: he was once the vice-president of the famous Golden Telecom and ROL — you may have started to go online with their Internet access cards. So, he also wanted to build a huge data center. And even managed to catch him in life.

What I am telling you all about is that the data center was built by a British entrepreneur with good European connections, so European customers immediately came inside. This means very clear reporting on all transactions, which gives some certainty that the processes and regulations within the data center are fully implemented.

The data center was built piece by piece, launching the machine hall at the machine hall. The first stage was at 220 pillars (they started building in 2012, and were put into operation in the middle of 2013). The second machine hall is 515 racks (1040 square meters, our first servers are there), and the third phase (where our core is) is 1100 racks (these are 2500 squares).

Three machines are independent of each other, but they are not three different data centers, because there is a cross-redundancy of part of the nodes. The first two machine halls are IBM L3 certified, the third one is uptime TIER III certified. In short, they are almost the same and the difference at the project level, for example, is that the UI needs dedicated parking for engineers, physical barriers for cars, somewhere 45 minutes of burning resistance (windows in the UI) against the hour (IBM), different sizes of doors and so on.



The master plan was made by the company Arup. The property has an Arup certificate for the best solution in the field of cooling and power supply. This is because the data center is commercial: the lower the PUE, the cheaper it is to exploit it. Therefore, they introduced everything in parts (that is, no equipment was idle, the engineer was purchased as it was scaled), and there was a struggle for every kilowatt. Now PUE is 1.15–1.4 depending on the season. The data center hosts quite a few Chinese customers, for whom with their approach to ecology and the use of electricity, these numbers are just space. They have the usual PUE can reach up to two.

At the time of the start of operation, the balance of customers they had about 50/50 domestic to foreign. At that time, everyone moved within the framework of the law on personal data protection and, as a result, on the Russian cloud infrastructure market (this is what we do) there was an increased demand for data centers (this is where cloud providers had to be located).

It is here that more and more companies from abroad are coming, because here stands a significant customer - Orange's foreign provider, which for many is both a vital reference point and a corporate standard. And the third machine building was financed by the Goldman Sachs Group company, which adds reliability to the project in the eyes of foreign managers. Data centers do not change every year, so it is important for them to think out 10 years ahead.

Given the abundance of external contacts, the support of the first line communicates in writing in Russian, English and Chinese. All inscriptions are duplicated in Chinese:



The data center, thanks to all the same Guy, is in two alliances of data centers - European and Asian. It's like airline alliances, when you can buy a ticket with a transfer between members of the alliance and you will be overloaded luggage without your participation. Very many need geo-distributed placements, and therefore such alliances are being made. The customer of one of the data center alliance actually works with the entire network in one window. In our market, this is still a unique phenomenon.

Physical protection

For PCI DSS objects, building protection is very important. The first perimeter is an external fence, which is equipped with sensors, cameras and barbed wire. At the entrance to the checkpoint. Then you need to get into the building itself (open by pass or camera), then you need to go through security (passes, inspection, removal of bottles with kefir), then you need to go to the machine room (biometric palm scanners, fingerprints or codes). In the machine room, the racks themselves are most often in fences (whoever puts them, usually also fingerprints or keys, there was one mantrap for the bank: this is a vestibule, where you can enter by pass or finger, and the second door - only by permission of the guard from the camera). Everywhere video surveillance.





Subjectively, the physical perimeter in DataPro is slightly stronger if you are standing without a fence. With the fence, everything here is much more interesting, because, for example, the Japanese are standing next to us. They have their own cell standard and a rod thickness of 6 mm. They say there are those who have 8 mm, but I did not measure. Many in the fences are infrared motion sensors or intersections, there are a couple of cells from floor to ceiling.



There are a lot of fences here. More than anywhere else in commercial data centers in Russia. At least from what I know.

Overview

Oddities begin immediately upon entry. Restroom for admins is more like a library:



You can look around a bit and see the player with real vinyl. There are vinyl Led Zeppelin and Deep Purple. Deepi shmot no.



That's because the investor loves rock even more than data centers. The same comrades make the festival Rockin'Russia. Cliff (who was the director before commissioning) wrote two songs for Scorpions (for example, One and One Is Three).

There is a bogilnitsa and a robber:





Next is a car that wets and warms the napkins:



And food with water (free):

Discharge area

Here is the preparation of equipment for skidding. Here, perhaps, the most interestingly equipped unloading zone among Russian data centers. First, it is big. Secondly, there is a tennis table. Thirdly, there is a special air cleaning. This is the acclimatization chamber for the first machine hall (we loaded the main equipment through another one, similar):



Pay attention to the height of the gate.

The equipment is brought in here, Karcher is vacuuming it (the usual model of the most powerful ones), then the iron in the boxes is settled, then it is released from the boxes (inside the cardboard is prohibited - and in general all fuel is prohibited).

Very comfortable stops with the possibility of mounting the door:

Providers

Generally, from the data center we need three things: proper cooling, proper communication and proper nutrition. About cooling, I will add that they have a zoo of conventional air conditioners, chiller-fan coil and free cooling in different rooms. But about communication it is very important that there is a direct connection with M-9 and there are peering nodes. That is, if the traffic runs from one data center server to another data center server (possibly another company) through the “nine” or similar European nodes, then the peering node will drive it directly through the local network. Well and further already peering providers. Given that there is one very large gaming company here, I guess who needed it first of all so that users did not have to lag.

At the site at the time of publication of the post 43 provider. A year ago, it was the second place in Russia in connectivity. Now, quite possibly, IXcellerate has already reached the first, but there has been no research so far.

On the roof there is a platform for satellite equipment, you can put any antennas, but so far nothing. They say one of the customers planned to put a satellite teleport there, but in the end did not.

Mashroom

Let's just go, and I'll show you what you can shoot.

Many cells:



Cooling optimization - fences and panels with removable slats:





Thermal sensors:



PDUs are made for them according to their project at a factory in the Moscow region. They say it is import substitution, which could. They, by the way, are locked:



Here is the big machine:













Here behind such doors technical premises and secret levels:



It happens that behind them are closed rooms for 25–35 racks for privacy. But it may be rechargeable. In the first machine room, conventional batteries, in the third already lithium-ion batteries. The latter are good with a separate telemetry of each battery, with a service life of up to 12 years (the lead has a maximum of 7, but rather, 5), they are more resistant to temperature and heat, plus two degrees do not knock a year out of service life. In the future, more than 5 years, they are cheaper than lead. Each battery has its own separate housing:



Here are the optical trays. Look at the care of people: these plugs prevent the cable from cutting against the edges of the holder:



This is gravitapa. I suggest to guess what it is:



In some cells, customers store stools, ladders, terminals. There are a couple of cabinets with spare parts. Requirement one - to from non-combustible material. In a pair of racks there are suitcases (these are placed on a sled and designed specifically for storing things in the rack). But the characteristic detail: metal pipes for slubotochki.



Our iron:

Nutrition

Two independent urban beams of 13 MW come to the territory.

Now food 2N. Next year there should be 3N - they are building an energy center for three gas turbines.

Here are diesels:



Concrete blocks, so that the machine does not even accidentally entered the diesel:



Near cooling:







And in the end we are waiting for a lawn for music festivals and football. There you can relax:

TSH data center

• Certificates: PCI DSS, ISO 27000 (27001), ISO 9000 (9001), ISO 14001: 2004.
• The building has an area of ​​6 thousand square meters, 1835 stands, capacity - 13.7 MW.
• Energy - 2N, UPS + DGU, cooling - chillers with free cooling, reserve - N + 1.

For now. The data center employs 50 people, including non-engineering staff. We chose a second site for a very long time and seemed to have chosen correctly (at least, our equipment has been here for more than a year). But once again I will note: if something changes, I will keep up to date.