Microsoft data centers
The first data center (data center) Microsoft was built in 1989. It was the first generation data center, now it is the fourth generation data center. Microsoft independently carries out the design, construction and management of data centers. For these tasks, we even have a special unit - the Global Foundation Services (GFS).
What tasks does Microsoft use for data centers?
First, to ensure the health of services that use millions of users around the world (XBOX Live, Hotmail, MSN, Zune, SkyDrive, etc.). Now it can be called the well-known term SaaS (Software as Service). Second, to provide a global scalable platform on the basis of which developers can create their own SaaS. This platform is Windows Azure, which implements the PaaS (Platform as Service) model. In facts and figures, it looks like this:
Basic data center architecture
Let's start with the basic architecture of the data center, it will help to understand the differences between the first and fourth generation data centers. Traditional data center infrastructure can be represented as follows (source of The Green Grid ):
')
Power supply system . The power supply system is responsible for powering the data center energy, usually tens of megawatts (MW). Usually (just in case) for data centers, two substations are used that supply electricity. From the substation ( Substation ) to the input of the data center, electricity flows with a voltage greater than the server ( IT Load ) is ready to accept to the input. Therefore, transformers are needed (Transformer), which lower the voltage.
In case of failure of the main power supply system, backup generators should be provided. In addition, industrial uninterruptible power supplies ( UPS ) are required. Notice that the black lines (electricity from the power plant) and the brown lines (electricity from the generators) go to the UPS in the figure, and the green lines (already passed through the UPS filter) come out. It is the “green lines” that go to the equipment of the data center. The simplest type of UPS is UPS based on batteries (backup battery UPS), but for data centers, this is usually not very suitable, because provides a fairly small reserve and duration of work. In our data centers we use diesel generators with a capacity of 2-2.5 MW. For example, such a generator can serve a village of 250 houses.
Cooling system The cooling system is responsible for cooling the equipment. The principle of its work is quite simple:
When the room is cooled it is very important to make sure that the hot and cold air does not mix. For example, compare four variations of the system:
The last fourth option is not used very often for racks with a high density of servers.
Total, gets that when building data centers it is necessary to solve the following tasks:
Technical details, such as the operation of the cooling system and generators, can be found in habrostats (from the Beeline block):
Energy Efficiency of the Data Center (Power Usage Effectiveness, PUE)
To determine the effectiveness of data centers, there is a special indicator (analogue efficiency) - Power Usage Effectiveness (PUE).
Traditionally, the PUE value is defined as the ratio of the total power consumption of the data center to the power consumption of only the useful IT equipment (server, storage, etc.) located in it.
Suppose that 15 MW is required for the operation of a data center (server, lighting, etc.), and only 10 MW is consumed by servers, therefore PUE = 1.5. It turns out that the lower the PUE, the better. Now 1.3-1.5 PUE is considered standard for TSODA. The fourth-generation data centers have a significantly lower PUE index than the first-generation data centers.
Data Center and ITPAC Generations (IT Preassembled Components)
The first generation is servers, the second generation is server racks, the third generation is containers, and the fourth generation is special modules (IT Preassembled Components, ITPAC).
In usual container external water cooling is used for what it is necessary to connect lines with cold water; In addition, their power efficiency is somewhat lower. For the newest modular data centers based on ITPAC, an air-water system is used ( spraying water droplets inside the module, which increases the rate of cooling through evaporation ), and the energy efficiency indicator (PUE) is 1.05, that is, for every watt sent to server power, just 0.05 watts is spent on the needs of the cooling infrastructure.
In fact, ITPAC is some standard for a complete module for building a data center. It contains specifications for the container and its components: the rack, the ventilation system and other equipment, as well as the supplied communications, including water, electricity and data network. An important feature is the possibility of using external air without prior cooling, the temperature of which can vary over a fairly wide range. The functioning of the internal air conditioning system is regulated depending on the temperature of the intake air. So, in the case of a significant increase in it, water cooling is turned on inside the ITPAC container, while reducing to a certain threshold, the air heated by the servers is used to maintain a comfortable environment inside the module. At normal temperatures, the intake air flow is sufficient to cool the servers. ITPAC can contain more than 2000 servers.
Building a data center using ITPAC is many times faster and more efficient: an economical cooling system, ITPAC containers can be manufactured remotely and delivered as assembled to the site under the data center.
Where are the data centers?
Below is the geographical presence of data centers. If we talk about Windows Azure, now it’s 8 data centers (4 in North America, 2 in Europe and 2 in Asia). By the way, two of them began to support Windows Azure not so long ago.
For Windows Azure, it is worth noting here that for data centers in neighboring regions they automatically and free of charge make copies of file storage data, i.e. this is automatic replication. As a result, we get 6 copies of our data (3 replicas for each data center). All these measures (replication, separate power supply, etc.) are aimed at improving the resiliency of the final developed application or service that we provide or that was developed on our PaaS platform. By the way, another interesting point: as we said earlier, for servers and racks you need either UPS or generators. Generators consume much more data center energy, so using UPS is more “economical”, but as we remember, its reserve may not be enough - then the server will fail or data will be lost (and this is not good). So, it is the approach to replication and geo-distribution that allows us not to lose data and transfer the SLA care from the infrastructure of the data center to the application and service levels, and this allows lowering the PUE ratio and increasing efficiency.
The data center in California looks like the figure below:
To feel the scale, let's compare with the football field (yellow rectangles, the area of ​​each of which is 2464 m2):
What's next? The fifth-generation data center?
What will be the fifth generation of data centers? Let's look at the data as a new form of energy:
Old day : era to data centers and SaaS, at home or businesses received electricity directly and used it at their discretion (for their server, etc.). What is the disadvantage here? A big "waste" of energy "along the way."
Today : with the advent of data centers, most of the work and calculations began to be done in them, and at the output we get data (SaaS). You can see that the "waste" of energy is already much less.
The Future : combine data center and energy source (substation). And this is a new way, because About $ 2 million is spent on the distribution of electricity per mile. The optical channel (and not the tower) is much cheaper. And we are already starting to follow this concept, for example, a new data center under construction in Ireland. The main distinctive feature of this data center will be the availability of its own infrastructure for managing power supply . I believe that many cloud providers have already thought about this.
What will happen next - the future will show, but new levels of energy efficiency have already been achieved through the use of advanced technologies and PaaS architecture.
If anyone is interested in the Microsoft data center infrastructure, then go to the Global Foundation Center (GFS) site - we have something to tell, because Microsoft is not only a software company, but also has a lot of experience in the development of data center modules and the data center architecture as a whole. On the video page you can go through a virtual tour of the data center, starting from the moment of its construction (it is even more interesting than the finished data center).
Ps. I want to thank Konstantin Kichinsky and Anzhdey Arshavsky for their help in preparing the article.
What tasks does Microsoft use for data centers?
First, to ensure the health of services that use millions of users around the world (XBOX Live, Hotmail, MSN, Zune, SkyDrive, etc.). Now it can be called the well-known term SaaS (Software as Service). Second, to provide a global scalable platform on the basis of which developers can create their own SaaS. This platform is Windows Azure, which implements the PaaS (Platform as Service) model. In facts and figures, it looks like this:
Basic data center architecture
Let's start with the basic architecture of the data center, it will help to understand the differences between the first and fourth generation data centers. Traditional data center infrastructure can be represented as follows (source of The Green Grid ):
')
Power supply system . The power supply system is responsible for powering the data center energy, usually tens of megawatts (MW). Usually (just in case) for data centers, two substations are used that supply electricity. From the substation ( Substation ) to the input of the data center, electricity flows with a voltage greater than the server ( IT Load ) is ready to accept to the input. Therefore, transformers are needed (Transformer), which lower the voltage.
In case of failure of the main power supply system, backup generators should be provided. In addition, industrial uninterruptible power supplies ( UPS ) are required. Notice that the black lines (electricity from the power plant) and the brown lines (electricity from the generators) go to the UPS in the figure, and the green lines (already passed through the UPS filter) come out. It is the “green lines” that go to the equipment of the data center. The simplest type of UPS is UPS based on batteries (backup battery UPS), but for data centers, this is usually not very suitable, because provides a fairly small reserve and duration of work. In our data centers we use diesel generators with a capacity of 2-2.5 MW. For example, such a generator can serve a village of 250 houses.
Cooling system The cooling system is responsible for cooling the equipment. The principle of its work is quite simple:
- At the entrance comes water (“blue lines”), which is cooled using cooling towers or cooling towers. Cooling towers are usually used in warm climates. For a cold climate, they are just trying to use outside cold air (such systems are much cheaper than cooling towers), but for this, low humidity should be outside and the air temperature should not exceed 23 degrees. Such systems are called adiabatic cooling systems ( Adiabatic System ), they are used in fourth-generation data centers.
- In the next step, large walkers or chillers (chiller) cool the water.
- Then the water enters the special conditioners - CRAC (computer room air conditioning unit), which are already in the data center (Zone 1).
- Air heated from the air (“red lines”) again goes through the entire cooling cycle.
When the room is cooled it is very important to make sure that the hot and cold air does not mix. For example, compare four variations of the system:
The last fourth option is not used very often for racks with a high density of servers.
Total, gets that when building data centers it is necessary to solve the following tasks:
- Give a maximum of the incoming energy directly to the servers, and not to other installations (cooling, air conditioning, etc.).
- Cooling data centers as efficiently as possible (the fewer installations and, consequently, the power used for this, the better).
- Make the most efficient use of data center space (with effective cooling, the density of servers can be significantly increased, and therefore the area and data center cost and area cost can be reduced).
Technical details, such as the operation of the cooling system and generators, can be found in habrostats (from the Beeline block):
- Datacenter in Yaroslavl: the realization of the dream of the last five years ;
- Data center in Yaroslavl: engineering solutions for smooth operation ;
- Building a data center, part 3: serve chilled .
Energy Efficiency of the Data Center (Power Usage Effectiveness, PUE)
To determine the effectiveness of data centers, there is a special indicator (analogue efficiency) - Power Usage Effectiveness (PUE).
Traditionally, the PUE value is defined as the ratio of the total power consumption of the data center to the power consumption of only the useful IT equipment (server, storage, etc.) located in it.
Suppose that 15 MW is required for the operation of a data center (server, lighting, etc.), and only 10 MW is consumed by servers, therefore PUE = 1.5. It turns out that the lower the PUE, the better. Now 1.3-1.5 PUE is considered standard for TSODA. The fourth-generation data centers have a significantly lower PUE index than the first-generation data centers.
Data Center and ITPAC Generations (IT Preassembled Components)
The first generation is servers, the second generation is server racks, the third generation is containers, and the fourth generation is special modules (IT Preassembled Components, ITPAC).
In usual container external water cooling is used for what it is necessary to connect lines with cold water; In addition, their power efficiency is somewhat lower. For the newest modular data centers based on ITPAC, an air-water system is used ( spraying water droplets inside the module, which increases the rate of cooling through evaporation ), and the energy efficiency indicator (PUE) is 1.05, that is, for every watt sent to server power, just 0.05 watts is spent on the needs of the cooling infrastructure.
In fact, ITPAC is some standard for a complete module for building a data center. It contains specifications for the container and its components: the rack, the ventilation system and other equipment, as well as the supplied communications, including water, electricity and data network. An important feature is the possibility of using external air without prior cooling, the temperature of which can vary over a fairly wide range. The functioning of the internal air conditioning system is regulated depending on the temperature of the intake air. So, in the case of a significant increase in it, water cooling is turned on inside the ITPAC container, while reducing to a certain threshold, the air heated by the servers is used to maintain a comfortable environment inside the module. At normal temperatures, the intake air flow is sufficient to cool the servers. ITPAC can contain more than 2000 servers.
Building a data center using ITPAC is many times faster and more efficient: an economical cooling system, ITPAC containers can be manufactured remotely and delivered as assembled to the site under the data center.
Where are the data centers?
Below is the geographical presence of data centers. If we talk about Windows Azure, now it’s 8 data centers (4 in North America, 2 in Europe and 2 in Asia). By the way, two of them began to support Windows Azure not so long ago.
For Windows Azure, it is worth noting here that for data centers in neighboring regions they automatically and free of charge make copies of file storage data, i.e. this is automatic replication. As a result, we get 6 copies of our data (3 replicas for each data center). All these measures (replication, separate power supply, etc.) are aimed at improving the resiliency of the final developed application or service that we provide or that was developed on our PaaS platform. By the way, another interesting point: as we said earlier, for servers and racks you need either UPS or generators. Generators consume much more data center energy, so using UPS is more “economical”, but as we remember, its reserve may not be enough - then the server will fail or data will be lost (and this is not good). So, it is the approach to replication and geo-distribution that allows us not to lose data and transfer the SLA care from the infrastructure of the data center to the application and service levels, and this allows lowering the PUE ratio and increasing efficiency.
The data center in California looks like the figure below:
To feel the scale, let's compare with the football field (yellow rectangles, the area of ​​each of which is 2464 m2):
What's next? The fifth-generation data center?
What will be the fifth generation of data centers? Let's look at the data as a new form of energy:
Old day : era to data centers and SaaS, at home or businesses received electricity directly and used it at their discretion (for their server, etc.). What is the disadvantage here? A big "waste" of energy "along the way."
Today : with the advent of data centers, most of the work and calculations began to be done in them, and at the output we get data (SaaS). You can see that the "waste" of energy is already much less.
The Future : combine data center and energy source (substation). And this is a new way, because About $ 2 million is spent on the distribution of electricity per mile. The optical channel (and not the tower) is much cheaper. And we are already starting to follow this concept, for example, a new data center under construction in Ireland. The main distinctive feature of this data center will be the availability of its own infrastructure for managing power supply . I believe that many cloud providers have already thought about this.
What will happen next - the future will show, but new levels of energy efficiency have already been achieved through the use of advanced technologies and PaaS architecture.
If anyone is interested in the Microsoft data center infrastructure, then go to the Global Foundation Center (GFS) site - we have something to tell, because Microsoft is not only a software company, but also has a lot of experience in the development of data center modules and the data center architecture as a whole. On the video page you can go through a virtual tour of the data center, starting from the moment of its construction (it is even more interesting than the finished data center).
Ps. I want to thank Konstantin Kichinsky and Anzhdey Arshavsky for their help in preparing the article.
Source: https://habr.com/ru/post/142612/
All Articles