How Yandex built a data center from scratch
This spring we received permission to operate our new data center. The first, for whom everything, even the building, was designed and built by the Yandex team from scratch. For the 18 years that people have been searching on the Internet by Yandex, we have come a long way from the server under the table of one of our developers to building a data center, where we use our own equipment. On the way, we had several data centers in Russia, into which we rebuilt the factories and workshops that had once stopped their work.
When we chose a place where you can build a data center from scratch, the cold climate was one of the most important factors. But it was in the course of this construction that we found a technological solution that allows us to operate the DC in a warmer climate. Now we are planning to build our next data center in the Vladimir region. And now we have all the opportunities to create a data center in Russia, which will become one of the most advanced in the world.
')
In this post we want to tell how we designed the DC, what difficulties our team encountered during the construction process, how the commissioning proceeded, what are the features of Yandex data centers, and how heat recovery, which you could already hear, was arranged.
As we have already said , the architecture of the company's services allows organizing redundancy at the level of data centers, rather than subsystems, which gives greater flexibility in choosing solutions and reduces the redundancy of systems and associated costs.
The data center and server architecture are designed as a single system, therefore, the operating parameters of engineering systems may differ from conventional ones - in particular, alternative uninterruptible power supplies can be used and the cumbersome cooling solutions can be abandoned due to the expansion of the allowable temperature range. All this leads to increased efficiency and lower total cost of ownership of infrastructure.
One of the metrics of data center efficiency is PUE . However, speaking of our DC, we are not going to put it at the forefront. First, we have no reason to measure PUE with our colleagues, and second, the existing differences in the method of measuring this parameter do not allow a comparison even between our own DCs.
However, the very essence of the PUE parameter is extremely important for Yandex. Commercial data centers can afford not to skimp on construction and equipment, knowing that in the end the client will pay for everything. In our case, we are our own client, so the low total cost of ownership of DC (TCO) is perhaps the most important criterion when choosing solutions in a company.
And of course, our team wants not only to increase the area for the installation of new racks, but also to feel professional pride in what we are doing.
Prior to the design of the Finnish data center, we managed to build (and partially close) 10 data centers. During the construction of the next DC, each time we tried to use the most interesting solutions that appeared on the market at that moment. This led to a wide variety of implementations, but now it allows us to choose the best solutions not on the basis of advertising catalogs, but using the experience gained in the company and the results of our own experiments.
The basic idea behind the Finnish DC project is simple: it is the use of direct free cooling 100% of the time. Direct freecooling (server cooling with outside air) makes it possible to abandon the use of not only heat exchangers with chilled water or other suitable coolant, but also simpler devices, like the Kyoto wheel , which stands in one of our DCs. In the new data center, we take the air from the street, run it through the servers and release it on the street, simultaneously selecting a part for mixing in the cold season. Our server hardware today is quietly working not only in the range of 20-24 degrees, which was recently the very “ASHRAE window”, but also at higher and lower temperatures. Therefore, there is no need to install capricious and expensive cooling systems, you can simply choose a region in which the average annual temperature is lower and spray water on the hottest days to lower the intake air temperature by a few degrees — this is called adiabatic cooling.
Adiabatic chamber and drift eliminators behind it
Those of you who have studied the design of DC, Facebook, which has been repeatedly published on the Internet, will find many analogies. But we must immediately stipulate that this is only a coincidence. When the concept of our DC was ready, we had a chance to visit one of the DC social networks, and we were surprised to learn that our colleagues had just implemented a project that was almost identical to ours.
Of course, the specifics of Oregon and Finland influenced the differences, but the principle of blowing air from the street is so simple and obvious that there is nothing surprising in the similarity of concepts. On the other hand, we noted with pleasure that in some ways our project will be even more interesting.
Diesel generators and dynamic ups
For example, a less complex adiabatic cooling system and the use of dynamic rotary UPS (DRIB) instead of a combination of diesel and classic battery UPS.
As can be seen from the illustration, cold air enters from the street (on the right side of the picture) into the mixing chambers, from where, after filtration, the fan walls are forced through the humidification module down into the cold corridors.
First row of filters
Second row of filters and intake fans
From the hot corridors the air enters the space of the second floor, which plays the role of a kind of buffer, from where a part goes to the mixing chambers.
The mixing chamber of external and exhaust air to achieve the optimum temperature
Hot air that is not used in mixing by exhaust fans is discharged from the other side of the building.
Inside the "hot corridor"
The solution is really very simple, but it involves the use of equipment that can operate at an appropriate temperature and humidity, which is difficult to achieve for commercial data centers.
Servers in work
As the first operating experience has shown, we still have considerable potential for improvement. We realized that it is possible to simplify the constructive building, completely abandon the use of water in the DC and, finally, expand the geography of construction up to the central regions of Russia. Thanks to this, we will be able to build our next data center in the Vladimir region, whose climate is much warmer than the Finnish one.
This question is the second most popular question we have to answer. The answer is so simple that it is even a bit awkward to talk about it.
The fact is that the unofficial talisman of Yandex is an elk (“Our elk!”). There are two countries all over the world in which these animals are valued and cherished - Sweden and Finland. But since Facebook has already been built in Sweden, we still have our northern neighbors. From the whole country we chose a city, on the arms of which there was an elk. So the die was cast, and we decided to build in Mäntsälä.
However, one moose is still not enough. The most important criteria by which selection was made at that time were the available electrical power (Finland’s highly stable power grid was built many years ago for the pulp and paper mill, which are sensitive to the quality of the supply voltage), good accessibility of optical lines, engineers eager for work and, of course, the cold northern climate , in terms of which it was possible to abandon the chillers and honestly use your favorite free cooling.
It should be noted that working with local partners was a real pleasure. Thus, a construction permit is obtained within one month (sic!), And the road, water, heat and power lines arise on the borders of the site in an incredibly short time.
In addition, we have implemented with the local authorities, so to speak, for the soul, a project for heat recovery. Having installed heat exchangers in the place of exhaust hot air, we heat the coolant and transfer it to the urban heating system. For the municipality to bring the water heated in this way with heat pumps to the required temperature turned out to be cheaper than building an additional boiler house for new districts of the city.
Actually heat exchangers
We presented this solution in March at CeBit (this year combined with Datacenter Dynamics ), and it attracted great attention - the subject of energy efficient technologies is very relevant today. On May 27, Mäntsälä won the Best Heat Pump City in Europe prize.
If you do not touch the content of the documentation, then already in the process of designing we faced with some interesting features of work in Finland.
There is no such rigid formalization of the design and composition of documents to which we are used. Speaking quite simply, the documentation specification should be exactly the same so that the contractor could get all the necessary information. To some extent, facilitated communication with the licensing authorities, which expressed their approval, based on how they understood the document, without paying attention to the completeness and design.
Under the roof where air intake takes place
Some of the representatives of the authorities did not seek to express their personal opinion about the project. The fire inspector, for example, was satisfied if the submitted package of documents contained a visa for a certified consulting company. Such a signature meant for him that the project had already been verified and complied with all local regulations.
On the other hand, the majority of small contractors refused to even estimate the cost of the work if the drawings did not state everything to the last screw. Such an approach puzzled not only us, but also our partners from other countries. As a result, many drawings contain much more information than we would expect to see.
When implementing the project, we used a three-dimensional model of the building, in which all the equipment was placed, cable routes were carried out, etc. It took some time to get used to this tool, but then it was very convenient to work with the model - it was possible to quickly solve some problems with mixing systems that were not immediately obvious in a two-dimensional representation.
Also, our colleagues taught us to use the online project to collect comments on the project, so that one could be sure that the comments would not be lost and would be regularly viewed at project meetings.
On the extraordinary adventures of Russian customers in Finland could write a separate large article. But the main conclusion is simple: the success of such projects is determined by the ability to overcome differences in the mentality and culture of the participants. And the greatest responsibility lies with us at the time of choosing a partner who can reconcile these differences. There are a lot of examples. The most amusing, perhaps, the following.
Start of construction
Finnish construction works from seven in the morning to three or four days. Imagine our bewilderment when the Russian team, returning from dinner, sees a friendly crowd of builders who have already completed their work shift today. The remaining half a day we see an almost deserted construction site, and against this background, the statements of the Finnish partners look a little strange, complaining that they have no time to meet the deadlines, although they are trying very hard.
Construction of steel structures
At the level of an ordinary worker, Finnish construction quality is not too different from what we are used to. A significant increase in quality occurs when adding several levels of professional foremen, supervisors and managers. And although the price of the “managerial component” is a shock, the result is very good.
And of course, it is impossible not to mention the legendary Finnish July, when the whole country quits work and begins to rest. In some companies it makes no sense to even call - in the office there is not a single soul. Since we are accustomed to consider these rare fine days as the ideal time to complete many construction works, this attitude on the part of contractors seemed to us like frank sabotage.
The frame is ready
But the most important lesson of this construction was the realization that projects of this scale require a very special approach. The way you organize a construction site has a disproportionately large effect on the outcome of the project than what you build. We saw that in another country people work differently. This does not mean that they are doing something bad - just different. It seems that we managed to understand each other, and most importantly, we got an excellent result together.
Perhaps the main thing that distinguishes this project from all the previous ones is an understanding from the very beginning of the importance of a full-scale commissioning (NDP). For high-quality commissioning of the object, and therefore, to minimize problems during operation, we still had a whole phase of the project called Commissioning during the design period. Reluctantly, we gave Commissioning two months, knowing that the commissioning of a ready-made DC will be postponed to these two months, and at the same time realizing that it will pay off in the future. Indeed, a number of flaws were found during the commissioning process. Nothing serious, but they would have brought additional trouble to the maintenance service.
Not only Russian, but also international experience shows that commissioning of a DC is still an unusual phenomenon. At best, disparate suppliers bring individual test programs, and then the operation group, together with the general contractor, drives off five or six of the most critical situations, usually associated with starting diesel engines and switching inputs.
A full description of the process deserves a separate story, so here we only mention that we have planned and carried out all five steps of the NDP with careful documentation of the results.
Commissioning performed a specially trained team. Formally, they were employees of the general contractor, but in fact acted independently, assessing the quality of work, including their colleagues. The operation service already formed by that time also took a huge part in commissioning. Naturally, representatives of the Moscow office were closely following the test results.
This approach was relatively new to us. We appreciated its capabilities and now we don’t even imagine how to start without it.
Interestingly, we started the recruitment and training of the maintenance service long before the installation of engineering systems. This allowed us to spend more time searching for really good specialists, to partially attract them even for the design period: unfortunately, this led to a change in the project at the last moment, but it reduced the project’s isolation from the needs of living people who will spend this DC more than one year. At some stages of commissioning, the operational service performed testing on its own under the supervision of the responsible party.
Entrance to the office
In Finland, much attention is paid to the issues of certification of specialists - it is simply illegal to perform certain types of work without proven qualifications. The second serious moment is the observance of the norms of labor protection and ecology. Therefore, in our team, the corresponding specialist appeared second in a row, and we could be sure that we would not violate strict legal norms even by accident. Also, a significant time lag allowed us to carry out all the required training and certification even before the start of operation, that is, in general “on the job”.
Back side of the data center building
In general, I would like to note the great interest of future attendants in their work. We are confident in the team that we scored, despite the fact that the data center, especially at this level, was seen by most of the guys for the first time.
It should be noted that now the Department of Operation Yandex is becoming a truly international team. This will not only allow the company to continue developing new teamwork skills, but will also oblige to raise the bar for an existing service.
Next we have plans to expand the already built sites, as well as the construction of a new data center, which promises to be even more progressive. Our goal is to make it one of the best in the world.
When we chose a place where you can build a data center from scratch, the cold climate was one of the most important factors. But it was in the course of this construction that we found a technological solution that allows us to operate the DC in a warmer climate. Now we are planning to build our next data center in the Vladimir region. And now we have all the opportunities to create a data center in Russia, which will become one of the most advanced in the world.
')
In this post we want to tell how we designed the DC, what difficulties our team encountered during the construction process, how the commissioning proceeded, what are the features of Yandex data centers, and how heat recovery, which you could already hear, was arranged.
Need
As we have already said , the architecture of the company's services allows organizing redundancy at the level of data centers, rather than subsystems, which gives greater flexibility in choosing solutions and reduces the redundancy of systems and associated costs.
The data center and server architecture are designed as a single system, therefore, the operating parameters of engineering systems may differ from conventional ones - in particular, alternative uninterruptible power supplies can be used and the cumbersome cooling solutions can be abandoned due to the expansion of the allowable temperature range. All this leads to increased efficiency and lower total cost of ownership of infrastructure.
One of the metrics of data center efficiency is PUE . However, speaking of our DC, we are not going to put it at the forefront. First, we have no reason to measure PUE with our colleagues, and second, the existing differences in the method of measuring this parameter do not allow a comparison even between our own DCs.
However, the very essence of the PUE parameter is extremely important for Yandex. Commercial data centers can afford not to skimp on construction and equipment, knowing that in the end the client will pay for everything. In our case, we are our own client, so the low total cost of ownership of DC (TCO) is perhaps the most important criterion when choosing solutions in a company.
And of course, our team wants not only to increase the area for the installation of new racks, but also to feel professional pride in what we are doing.
Idea
Prior to the design of the Finnish data center, we managed to build (and partially close) 10 data centers. During the construction of the next DC, each time we tried to use the most interesting solutions that appeared on the market at that moment. This led to a wide variety of implementations, but now it allows us to choose the best solutions not on the basis of advertising catalogs, but using the experience gained in the company and the results of our own experiments.
The basic idea behind the Finnish DC project is simple: it is the use of direct free cooling 100% of the time. Direct freecooling (server cooling with outside air) makes it possible to abandon the use of not only heat exchangers with chilled water or other suitable coolant, but also simpler devices, like the Kyoto wheel , which stands in one of our DCs. In the new data center, we take the air from the street, run it through the servers and release it on the street, simultaneously selecting a part for mixing in the cold season. Our server hardware today is quietly working not only in the range of 20-24 degrees, which was recently the very “ASHRAE window”, but also at higher and lower temperatures. Therefore, there is no need to install capricious and expensive cooling systems, you can simply choose a region in which the average annual temperature is lower and spray water on the hottest days to lower the intake air temperature by a few degrees — this is called adiabatic cooling.
Adiabatic chamber and drift eliminators behind it
Those of you who have studied the design of DC, Facebook, which has been repeatedly published on the Internet, will find many analogies. But we must immediately stipulate that this is only a coincidence. When the concept of our DC was ready, we had a chance to visit one of the DC social networks, and we were surprised to learn that our colleagues had just implemented a project that was almost identical to ours.
Of course, the specifics of Oregon and Finland influenced the differences, but the principle of blowing air from the street is so simple and obvious that there is nothing surprising in the similarity of concepts. On the other hand, we noted with pleasure that in some ways our project will be even more interesting.
Diesel generators and dynamic ups
For example, a less complex adiabatic cooling system and the use of dynamic rotary UPS (DRIB) instead of a combination of diesel and classic battery UPS.
As can be seen from the illustration, cold air enters from the street (on the right side of the picture) into the mixing chambers, from where, after filtration, the fan walls are forced through the humidification module down into the cold corridors.
First row of filters
Second row of filters and intake fans
From the hot corridors the air enters the space of the second floor, which plays the role of a kind of buffer, from where a part goes to the mixing chambers.
The mixing chamber of external and exhaust air to achieve the optimum temperature
Hot air that is not used in mixing by exhaust fans is discharged from the other side of the building.
Inside the "hot corridor"
The solution is really very simple, but it involves the use of equipment that can operate at an appropriate temperature and humidity, which is difficult to achieve for commercial data centers.
Servers in work
As the first operating experience has shown, we still have considerable potential for improvement. We realized that it is possible to simplify the constructive building, completely abandon the use of water in the DC and, finally, expand the geography of construction up to the central regions of Russia. Thanks to this, we will be able to build our next data center in the Vladimir region, whose climate is much warmer than the Finnish one.
Why Finland?
This question is the second most popular question we have to answer. The answer is so simple that it is even a bit awkward to talk about it.
The fact is that the unofficial talisman of Yandex is an elk (“Our elk!”). There are two countries all over the world in which these animals are valued and cherished - Sweden and Finland. But since Facebook has already been built in Sweden, we still have our northern neighbors. From the whole country we chose a city, on the arms of which there was an elk. So the die was cast, and we decided to build in Mäntsälä.
However, one moose is still not enough. The most important criteria by which selection was made at that time were the available electrical power (Finland’s highly stable power grid was built many years ago for the pulp and paper mill, which are sensitive to the quality of the supply voltage), good accessibility of optical lines, engineers eager for work and, of course, the cold northern climate , in terms of which it was possible to abandon the chillers and honestly use your favorite free cooling.
It should be noted that working with local partners was a real pleasure. Thus, a construction permit is obtained within one month (sic!), And the road, water, heat and power lines arise on the borders of the site in an incredibly short time.
In addition, we have implemented with the local authorities, so to speak, for the soul, a project for heat recovery. Having installed heat exchangers in the place of exhaust hot air, we heat the coolant and transfer it to the urban heating system. For the municipality to bring the water heated in this way with heat pumps to the required temperature turned out to be cheaper than building an additional boiler house for new districts of the city.
Actually heat exchangers
We presented this solution in March at CeBit (this year combined with Datacenter Dynamics ), and it attracted great attention - the subject of energy efficient technologies is very relevant today. On May 27, Mäntsälä won the Best Heat Pump City in Europe prize.
Design
If you do not touch the content of the documentation, then already in the process of designing we faced with some interesting features of work in Finland.
There is no such rigid formalization of the design and composition of documents to which we are used. Speaking quite simply, the documentation specification should be exactly the same so that the contractor could get all the necessary information. To some extent, facilitated communication with the licensing authorities, which expressed their approval, based on how they understood the document, without paying attention to the completeness and design.
Under the roof where air intake takes place
Some of the representatives of the authorities did not seek to express their personal opinion about the project. The fire inspector, for example, was satisfied if the submitted package of documents contained a visa for a certified consulting company. Such a signature meant for him that the project had already been verified and complied with all local regulations.
On the other hand, the majority of small contractors refused to even estimate the cost of the work if the drawings did not state everything to the last screw. Such an approach puzzled not only us, but also our partners from other countries. As a result, many drawings contain much more information than we would expect to see.
When implementing the project, we used a three-dimensional model of the building, in which all the equipment was placed, cable routes were carried out, etc. It took some time to get used to this tool, but then it was very convenient to work with the model - it was possible to quickly solve some problems with mixing systems that were not immediately obvious in a two-dimensional representation.
Also, our colleagues taught us to use the online project to collect comments on the project, so that one could be sure that the comments would not be lost and would be regularly viewed at project meetings.
Building
On the extraordinary adventures of Russian customers in Finland could write a separate large article. But the main conclusion is simple: the success of such projects is determined by the ability to overcome differences in the mentality and culture of the participants. And the greatest responsibility lies with us at the time of choosing a partner who can reconcile these differences. There are a lot of examples. The most amusing, perhaps, the following.
Start of construction
Finnish construction works from seven in the morning to three or four days. Imagine our bewilderment when the Russian team, returning from dinner, sees a friendly crowd of builders who have already completed their work shift today. The remaining half a day we see an almost deserted construction site, and against this background, the statements of the Finnish partners look a little strange, complaining that they have no time to meet the deadlines, although they are trying very hard.
Construction of steel structures
At the level of an ordinary worker, Finnish construction quality is not too different from what we are used to. A significant increase in quality occurs when adding several levels of professional foremen, supervisors and managers. And although the price of the “managerial component” is a shock, the result is very good.
And of course, it is impossible not to mention the legendary Finnish July, when the whole country quits work and begins to rest. In some companies it makes no sense to even call - in the office there is not a single soul. Since we are accustomed to consider these rare fine days as the ideal time to complete many construction works, this attitude on the part of contractors seemed to us like frank sabotage.
The frame is ready
But the most important lesson of this construction was the realization that projects of this scale require a very special approach. The way you organize a construction site has a disproportionately large effect on the outcome of the project than what you build. We saw that in another country people work differently. This does not mean that they are doing something bad - just different. It seems that we managed to understand each other, and most importantly, we got an excellent result together.
Commissioning
Perhaps the main thing that distinguishes this project from all the previous ones is an understanding from the very beginning of the importance of a full-scale commissioning (NDP). For high-quality commissioning of the object, and therefore, to minimize problems during operation, we still had a whole phase of the project called Commissioning during the design period. Reluctantly, we gave Commissioning two months, knowing that the commissioning of a ready-made DC will be postponed to these two months, and at the same time realizing that it will pay off in the future. Indeed, a number of flaws were found during the commissioning process. Nothing serious, but they would have brought additional trouble to the maintenance service.
Not only Russian, but also international experience shows that commissioning of a DC is still an unusual phenomenon. At best, disparate suppliers bring individual test programs, and then the operation group, together with the general contractor, drives off five or six of the most critical situations, usually associated with starting diesel engines and switching inputs.
A full description of the process deserves a separate story, so here we only mention that we have planned and carried out all five steps of the NDP with careful documentation of the results.
Commissioning performed a specially trained team. Formally, they were employees of the general contractor, but in fact acted independently, assessing the quality of work, including their colleagues. The operation service already formed by that time also took a huge part in commissioning. Naturally, representatives of the Moscow office were closely following the test results.
This approach was relatively new to us. We appreciated its capabilities and now we don’t even imagine how to start without it.
Exploitation
Interestingly, we started the recruitment and training of the maintenance service long before the installation of engineering systems. This allowed us to spend more time searching for really good specialists, to partially attract them even for the design period: unfortunately, this led to a change in the project at the last moment, but it reduced the project’s isolation from the needs of living people who will spend this DC more than one year. At some stages of commissioning, the operational service performed testing on its own under the supervision of the responsible party.
Entrance to the office
In Finland, much attention is paid to the issues of certification of specialists - it is simply illegal to perform certain types of work without proven qualifications. The second serious moment is the observance of the norms of labor protection and ecology. Therefore, in our team, the corresponding specialist appeared second in a row, and we could be sure that we would not violate strict legal norms even by accident. Also, a significant time lag allowed us to carry out all the required training and certification even before the start of operation, that is, in general “on the job”.
Back side of the data center building
In general, I would like to note the great interest of future attendants in their work. We are confident in the team that we scored, despite the fact that the data center, especially at this level, was seen by most of the guys for the first time.
It should be noted that now the Department of Operation Yandex is becoming a truly international team. This will not only allow the company to continue developing new teamwork skills, but will also oblige to raise the bar for an existing service.
What's next?
Next we have plans to expand the already built sites, as well as the construction of a new data center, which promises to be even more progressive. Our goal is to make it one of the best in the world.
Source: https://habr.com/ru/post/258823/
All Articles