Race to Tallinn data centers

We have never noticed that all articles about data centers end up as a carbon copy with the same phrase - we are waiting for you and we will be very happy to see you, but only if a hard one brings you to our city N, and especially to its industrial zone or to a dark forest in the suburbs.

But there are in this world the right guys, who not only will be glad in words, but directly take and bring you to themselves to show everything as it is, without photoshop embellishment and articles of effective sales. Taking the direction to write about what I saw without boring numbers in the previous article , now, I hope, we will have a decent continuation.

This time for me, the heroes were representatives of the Estonian provider Infonet, who, in honor of the commissioning of the first line of their data center in Tallinn Infonet DC, together with partners from Comfortel organized a trip for colleagues from St. Petersburg and Moscow. And they did it extremely pleasantly: instead of praising themselves beloved all day, they agreed on visits to two more data centers, leaving the guests the opportunity to draw conclusions and write similar reports. By the way, it was planned to visit 4 datik, but the sad reasons for visiting only three are slightly lower.

So today I want to spend some of your time talking about the characteristics of the telecommunications market in Estonia in general and what Tallinn data centers are in particular.

Disposition or Estonian Internet without a joke and in the context

Estonia is a small country. By area, it is almost equal to the Moscow region, and the entire population does not reach half a million. It is not known for certain how, but something helps to keep this small country leading positions in various ratings of Internet penetration, level of development of IT technologies, etc.

As many probably know, in Estonia for several years now the so-called e-government has been functioning with confidence, which the countries of the rest of the world are guided by. In short, each person has his own unique ID card, completely replacing the classic analog signature on analog paper. With this card they vote, pay taxes and fines, keep medical history, sign contracts, buy real estate, prove their education and work experience, i.e. Absolutely all information about a citizen and his life activity is stored on the network and can be obtained by anyone with access to it. Full refusal of papers wherever possible.

The incredible convenience of the whole idea is obvious. But how could such a tiny and far from richest European country overtake the major powers, investing hundreds of millions of dollars in the development of such services?

Not the last role in this business was played by the fact that the ISP market in Estonia appeared a long time ago, therefore it is saturated and in fact for many years divided among the three main operators. And it is divided, and not sawn by cartel collusion, because this is strictly supervised by the supervisor. These providers are called Elisa, Telia and Infonet who invited us.

But simply the fact that the market is "closed" does not in itself mean anything. I thought that a high entrance threshold could hinder the appearance of new players on the market, but it turned out that rent of cable ducts costs ridiculous money - something around 6 euro cents per meter of cable. Getting technical conditions is like running off a loaf of bread, and the issuance of radio frequencies generally occurs as if in between cases. Probably an effective weapon of existing operators is the long-term cost optimization. The local market is almost 30 years old, everyone knows each other and can solve any problems in an incredibly efficient way. So, entry into the market of new players is actually blocked, although no one forbids trying. Therefore, the bulk of the closest competitors (if you can call them that at all) of the local big three are virtual operators providing telephony services and similar small things. But in fact, these operators are something like our home networks from the time of the birth of the RuNet, when connecting houses in the quarter with a slung air line was considered a sign of a serious approach to communication.

A natural question arises: what is the operator to do if dumping for the sake of luring new customers makes no sense, incomes are known and there is no concern for tomorrow? The answer is simple - to develop the infrastructure, its services and add new services. For example, you can create your own TV channel and sell it to cable networks or provide a complete package of hosting services. Or you can, as in the case of Infonet, build your data center.

By the way, about the sad circumstances of not visiting a single data center. From the trip fell datik owned by Telia. In fact, this is the Estonian equivalent of our Rostelecom, i.e. the direct heir of the Soviet telephone monopolist with all the resulting infrastructural and administrative capabilities, which does not prevent it from being a Finnish-Swedish company better known in the world as TeliaSonera. And so they didn’t give agreement to the last, delaying the decision in every possible way for various reasons. But on the sidelines, it was indicated that the group of signalers is 90% made up of Russian citizens and the state datum things are incompatible a little more than completely. These are sanctions here. Treat this as you wish.

Therefore, we end up with theoretical studies and proceed to the practical part - to the trip.

Let's hit the road

It was necessary to go to Tallinn from St. Petersburg, the distance between the cities was quite trifling (about 400 kilometers) and the choice of means of transportation fell on the train as the most comfortable one. Judge for yourself - along with customs, the road takes about 7 hours, you are taken away from the center of one city, brought to the center of another, but for now you can go chatting with colleagues and sleep in a soft one or even hold a meeting in a dining car. Also, affordable prices and half-empty cars on a permanent basis were added to the asset.

But the trouble came from where it was not expected - from schoolchildren. Or rather from school holidays. Driven by civic consciousness to the future of the country and parental duty to children, parents completely bought tickets for the flight we needed. And only when the realization of the scale of the tragedy came to us, an exceptionally happy coincidence and the position of the stars in the sky allowed us to purchase tickets for about half of the participants in the trip. For the rest, a small bus was chartered in panic mode, as all big ones were bought up too, guess who. Well, a few citizens had to go in cars, in the ranks of which I decided to join, because I have never crossed this part of the border in this way and it’s a sin not to get this experience.

Since, by the will of fate, our group was divided into three, it was necessary to choose a common gathering point in Tallinn. Without further ado this point was chosen office Infonet located in the city center. And here we had the first cultural shock.

Imagine that you make an appointment at the office of the operator among the top three in the country. What are your expectations? Probably you will look for a modern building of glass and concrete, branded car installers in the parking lot, huge logos and other attributes of wealthy telecoms. But obviously not such a one-story hut behind a lovely fence with a barely noticeable sign. Although it is not even a hut at all, but quite a monument of architecture, for a moment! Here, even the colors of paint are not chosen anyhow as, but are indicated by the finger of the body engaged in the protection of historical objects.

But appearances are deceptive, and within this grace of the grandmother's village a stylish 3-storey office is found. The administrative staff is located on the middle floor, the upper floor is left to the technical department, and the lower floor (the basement does not turn out to be called the language) is given to such strategically important things as a kitchen, a shower and a wall of glory, filled with cups for sporting achievements.

What kind of wall, you ask? Where does the provider have such a number of athletes? The answer was extremely unexpected, at least for us. In a divided market, as already mentioned, when a provider can be distracted by any nonsense, he can afford such a thing as a football club. And not some nominal sponsorship participation, but as a full-fledged business line. At the same time, this club is not a member of the district courtyard league, but one of the best clubs in the country, the winner of the highest league and, no matter how you look, local pride and the subject of adoration.

When the whole group got together and the coffee was drunk, it turned out that we would be moving around the city just by the FC Infonet bus. Bus - dump head. In a row of three comfortable seats, instead of the standard four, toilet, mini kitchen, Internet and other benefits of civilization. Apparently in Estonia really love their players.
But enough of the lyrics, let's talk about data centers!

Levira

Quite consciously, I didn’t even try to make any inquiries about the TTX data centers that I had to visit. Therefore, when we were unloaded from the bus at the Tallinn TV tower, we had to immediately turn to more knowledgeable comrades for informational support.

It turned out that Levira is another office resembling our Rostelecom, i.e. half of the state organization, which for historical reasons (it was founded more than 20 years ago) was the first to engage in what can now be attributed to telecoms: television, radio, media provider. I also want them to put Internet into an asset, but they turned their attention to this area much later and began to engage in as much as. Anyone who has experienced WiMAX technology can wipe away a tear.

Accordingly, the television tower belongs to them, which is why it is quite effectively used not only for its intended purpose. However, this is a normal scenario when an infrastructure facility is placed under private management.

On the approaches to the object, a flowered armored vehicle was discovered (in a hurry it was not imprinted), an exhibition on the Soviet-era life, various movie entertainment for tourists, a nice souvenir shop and an elevator to the observation deck were located on the first floor. But due to time constraints, we were decisively driven away from the magnificence of a bygone era and sent to basements.

Let's digress for a second and think about why and how to do the data center in the TV tower? The answer lies in the question itself. The TV tower is an object immediately designed for the needs of telecommunications, and in those times when the equipment was very large, heavy, it was very hot and consumed an unimaginable amount of energy. After the decline of the era of analog television, all this lamp magnificence was removed from the building, all unnecessary cables as thick as a foot were cut out, but the power and cable routes were not gone anywhere. So it means that by themselves we got the perfect conditions for the arrival of digital devices. Here are just in size they, as we all know, are ten times smaller than analog grandfathers. So, empty areas were added to idle capacities, and on the basis of all this, in 2012, a full-fledged data center was established. Now there are 9 of them.

If you have been to the oldest traffic exchange point in your city or in any old dat, then you can wonder what the legacy of the times of wild telecom is. Now the photographs of the racks, where the spiders from the wires do not see the equipment, it is customary to frighten young engineers, but not long ago no one followed this and did what it took.

Either in order to save from possible shock, or in order not to be ashamed, they did not take us to the old halls, but carefully showed us the new machine building. Literally from the threshold, he was surprised by the fact that inside was warmer than outside in the corridor. Explained this riddle for two reasons. Still, we are on several floors underground and the fact that it is cold in the corridor is normal and unchanged throughout the year. And second, in the new part of the data center, the equipment is adjusted according to the updated standards, according to which the temperature can be 23 degrees, and not 19 as before.

By the way, the cooling here is water, with the corridors turned inside out, which caused many questions. The photo above shows that every three racks there is a cooling module (it glows blue). Rather, it would be more correct to say “heat sink”. There are three such modules for a row of six cabinets: at the beginning, end, and one in the middle. The air between the racks is mixed in a closed internal corridor, so the installation of servers is from the outside, not from the inside of the cold corridor, as we all used to. Yes, and the cold corridor here performs the role of hot. The estimated cooling capacity of such a scheme: 7kW per cabinet. During the commissioning test, it turned out even more, but decided to leave a small reserve. Although the figure of 7 kW sounds pretty solid. By the way, for the first time I saw an interesting solution with the placement of blocks of sockets on the front of the servers. Despite the seeming illogicality, 21e cabinets allow you to connect everything conveniently, leaving the back panels free for inter-server cables.

In a word, the scheme is extremely unusual and causes many questions. But according to local engineers, the failure of one module does not affect the temperature in any way. The failure of the second will force the last living to work at the limit of possibilities, but for some time he will be able to maintain an acceptable temperature.

Another infrastructure feature was found in diesel engines. First, there are a lot of them here. Two stands somewhere in the basements, a few - on the street, and the reserve of rolling is in the foreseeable space. Secondly, because of the strategic value of the facility, exercises are held once a quarter, and the entire TV tower runs on diesel for 4 hours. Quite everything, including data center. So the classic problem “What if it does not start?” Is excluded here.

In general, the place turns out quite interesting. The walls, the floor, the ceiling — wherever you go, it is from three to five meters of concrete. The load on the mass here can withstand any. Of course, such a problem does not arise every day, but when it arises, where to run is absolutely incomprehensible. And then at least arrange a ballet with elephants.

But not having sustained the long oppression of the legacy of the damned communists, we bowed and advanced to the next point of our route.

Linx

Strictly speaking, this is no longer Linx. The site has recently been purchased, but out of habit, everyone calls it the former name.

An average Russian, accustomed to the scope of construction, will experience visible difficulties in finding this dataset. Imagine that they bring you to the usual street view, where, on the one hand, there are nice apartment buildings, a car wash, some kind of big laundry, a shop and other attributes of a sleeping area. But upon closer inspection, it turns out that what you took as the backdrop of a store is actually the entrance to the data center, it just so happens that the air conditioning units hang at the main entrance and the whole structure resembles a rear entrance to the store. But if no joke, this modest place is one of the most important in Estonia from the point of view of communication. Among communication engineers, this object is known as TLL-IX.

In fact, the story of this place begins quite funny. Initially there was a clearing, on which stood a booth meter for two. An unremarkable booth, only inside it, a submarine cable connecting Estonia, Finland and Sweden came out of the ground. But in some incomprehensible way, these comrades managed to go bankrupt, but the booth remained and the owners of Linx didn’t be stupid, bought the whole clearing, surrounded it with a fence and built a dashboard, which became one of the local IX nodes, popularly called the golden triangle. 200 meters one way and you come to the main state data center, 200 meters another and you are already at the Telia data center.

The logical result is that there are so many cable sewers under our feet that any person with something more than a shovel is perceived here as a personal enemy and a potential source of terrible misfortunes. There are channels to all neighbors: Helsinki, Riga, Stockholm, St. Petersburg.

Inside the data center is no less surprising than the outside - you open the door, and immediately start the server. To the question: “Where are the guards with machine guns hiding and how to turn off the minefields?” A concise answer was received: “This is not accepted here.” Although in reality everything around is covered with cameras and alarm sensors, and in which case specially trained guys come face down on the floor in a few minutes. As it turned out later, there is no even a shift of engineers here. There is no one on a permanent basis at all. There is work - a responsible person comes, there is no work - not a single living soul.

And this is also the first date for me, where they honestly say that they are certified as TIER II, because they have only one diesel generator. In all other respects, they correspond to the top three, and perhaps in the future they will be engaged in obtaining a new status. But they are certified to ISO 27100, which is a crucial parameter for all those who work with European clients and should undergo an international audit. It is stated that they were the first in Estonia to receive this certificate.

They have an interesting situation with power supply. I myself am far from electricians, but as more literate people explained, the “Soviet” voltage of 6kV is fed here. And according to European standards there should be 10kV and, of course, gradually the entire infrastructure is transferred to this standard. That's just under the "gradually" here it is necessary to understand tens of years. Therefore, there is only one transformer at 6kV, while InfonetDC immediately installed two 2-system transformers designed for 6 and 10 kV at once. And the reserve in general comes from ordinary urban networks without any special transformers.
But inside there is a really powerful IX, where all that is needed is surrounded by an additional fence, and there is equipment worth about a dozen Moscow apartments. Although now there is a replacement for a newer, and according to the glorious tradition, it decreases by several times. Ah, we lose the beauty of telecom equipment.

Due to direct access to highways, the backbone of customers is content providers. For example, the largest hosting provider in Estonia is located here and under this roof there are approximately 80% of the sites in the * .ee zone

And the cherry on the cake is the POP global provider of Level3, i.e. This is a number of racks (either 5, or 6). If someone does not know, this is one of the Tier-1 operators, and their AS3356 has the largest number of connections in the world. About the presence of all local and large European operators can not even speak.

Cooling is organized according to the classical scheme, but implemented in some strange way. From the ceiling protrude hefty air ducts, of which the air flow hits with such force that there is no desire to stand directly under it. Naturally nails to the floor. Knowing the love of Europeans to save resources due to high prices for electricity, I dare to suggest that this system is of the same age as the dater, and is waiting for its replacement. Although it is possible with such a volume, everyone is satisfied with its energy efficiency.

I was amused by the story about the local automated process control system, and in fact the main control board of the engineering systems of the building. Since the data center here is not the first year, the shield was assembled by craftsmen from local engineers who were led by a certain Russian comrade. By itself, the shield delivers exorbitant aesthetic pleasure, and even worked with virtually no complaints. That's just the subsequent story as in jokes: Comrade quit, no one really knows how it works, so I had to order a vendor solution. But they did not dare to dismantle the shield. Painfully beautiful.

Infonet dc

And so, when it became completely dark outside, we got to the last point of our journey. Like the previous two, this data center has its pronounced main feature. But it is not engineering here, but completely business. Judge for yourself: as I said at the beginning, the history of sites for accommodation here is many years old, so despite a good technological base, many have a problem of moral obsolescence. Something it reminds a joke about mice and cactus: you want it or not, but you need to place equipment somewhere. On the other hand, Infonet does not hide the fact that they will be very happy to see customers from Russia who, for one reason or another, need to place equipment outside the country. For example, they are ready to conduct all contractual work in Russian, and for convenience they have opened a branch in Russia, which eliminates an incredible number of problems that arise when it comes to working with a foreign company.

Well, okay, let's drop the lyrics and move on to purely practical things. The history of the building itself, where the center is located, is pretty wild by our standards. Initially this is the workshop of the former factory for the production of ceramic tiles. Rather, not even the shop, but only part of it. It turns out in Estonia you can easily buy a piece of a building like this. But it turned out that not everything is so simple, because the firemen, they are firefighters everywhere. Rather, it very well coincided with the desire to separate from their neighbors and the requirements for fire safety. It turned out that, according to local standards, the access to fire engines from any direction should be provided for such an object and no one is interested that your object is just a piece of the building. How to implement this requirement? The correct answer is to demolish the walls with a roof in the middle of the shop, so that you can get two from one huge room and enclose your part with a fence. And it seems that's the trouble, but stop! We ourselves wanted to physically separate from our neighbors and tear down some of the old structures.

After such simple manipulations we get our separate building, with a ring road, and even with a large lawn around the perimeter, since here with this strictly. Want to buy a piece of the industrial zone - smash around the lawn.

From the building itself left only the columns and supporting farms. Everything else was broken to zero and rebuilt, including huge I-beams. By the way, in the middle of the construction it was decided not to paint the columns, but to preserve all the rock paintings inherited from the natives. It seems like a trifle, and makes a nice variety in the grayness of the corridors.
We go inside and immediately find such a sweet heart to every Russian thing, like a checkpoint with turntables and a guard behind the glass. Frankly speaking, these are not at all those passageways we were used to, but after data, where all physical security is a door with a magnetic lock, there is just an incredible breakthrough in the field of physical security of equipment.

At the moment, the first line has been put into operation, i.e. one hall. There are four planned in total. Why four, not three or five? The answer is simple - the building is almost square, so why wonder, let's just divide it into 4 equal parts and start building it up. European practicality in all its glory.

In general, Infonet DC looks like a giant against the background of the local market. The area and the allocated power they have the largest in the country. The main thing is not to compare with the Russian scale and reality. Judge for yourself - 4 MW of input power will not surprise anyone here, but here it is the first such facility. : Tier-III, - , . TIER III ready .

. . , , .., , . , , , , , - , , .. . , , — , . — , , — Stulz CyberCool 2. , .
, ?

( ), - , . , , .

, . , .

FM200 , , , . Some time. , 125, , . , , , , , , .

, , — , .

- , . .
, — . , .

Total

- - — . - - , , , . — , . , . , . , . ...

, : , , , .

PS Olde Hansa on the main square. Very atmospheric, but very expensive, but very tasty. If you can persuade your toad to give a hundred euros for dinner for two, you will not regret. So my advice to you is not to negotiate with the toad, but to put in front of the fact that it is necessary!