How the cooling system of the NORD-4 data center was created
In late September, the project of the cooling system for the NORD-4 data center won the Russian Data Center Awards 2016 in the nomination “The best solution in the field of engineering systems” . Today we will tell in details how the cooling system of the largest data center in Russia was created.
Type of building. NORD-4 was immediately planned to be multi-storey: on the first floor of the data center there is an energy center, on the second and third - 8 rooms for 2016 counters.
Cooling capacity. To remove heat from IT equipment with a capacity of 9,000 kW and associated engineering infrastructure, at least 10,000 kW of cooling capacity was required.
Temperature conditions in the hall . In the cold corridors of the machine room, the temperature should be within 23–27 ° . In such conditions, the equipment works stably, and engineers in cold corridors do not catch cold.
')
Climatic conditions on the street. The outdoor temperature for the project is taken from building codes and regulations (SNiP). In the Moscow region, in winter the temperature drops to –42 ° , and in the warm season it reaches +37 ° .
Certification. The data center was planned to be certified according to the standards of Uptime Institute Tier III (project, finished building and operation processes). According to the standard in the Tier III data center, any component of the system can be turned off for maintenance without interruption. This means that each element has a reserve of at least N + 1.
Launch schedule. The data center was immediately built with a view to a phased launch, so all systems needed to be designed so that the installation and commissioning of new capacities did not interfere with the already operating machine halls.
Type of IT equipment. NORD-4 is a commercial data center: it will be filled gradually and with a variety of racks. This means that you can not rigidly fix the layout of the racks in the engine room and pre-isolate the hot or cold corridors.
Before we dwell on the chiller scheme, which was implemented, we considered several alternatives.
Direct expansion air conditioners (DX). This configuration is often called simply Freon. In the machine rooms are located cabinet freon air conditioners with evaporator, compressor, thermostatic valve. On the roof or near the data center building there are external blocks with a capacitor. Air conditioners and outdoor units are interconnected copper freonoprovodami.
In this scheme, the air conditioner cools the air directly in the engine room without an intermediate heat carrier (water, air).
The principle of operation of the freon cooling scheme.
This system works in 5 of our data centers, so we are well acquainted. Compared to the chiller scheme, it has a number of advantages: it is cheaper in capital investments, easier and faster to install. Since all air conditioners are autonomous, to reserve a system, it is enough to add the required number of extra air conditioners.
Despite these advantages, it was not suitable for the NORD-4 project due to the following restrictions:
Kyoto Cooling. This is a dual circuit with natural air cooling (free cooling). Data center air circulates in the inner loop, and outside air is supplied to the outer loop. In a heat exchanger-recuperator of the rotor type, or "wheel", the hot air from the IT equipment is cooled by outdoor air. Kyoto Cooling does not involve compressors (that same natural cooling) at low street temperatures.
The principle of operation of Kyoto Cooling.
This option bribed its energy efficiency: in the free-cooling mode, electricity is spent only on the drive of the electric motors of the fans. Data centers using this scheme can boast one of the lowest PUEs - 1.09–1.13. Unlike the chiller circuit, Kyoto Cooling works without water, so there is no risk of leakage.
At the same time, Kyoto Cooling has specific requirements for building design:
"Wheel" with a horizontal axis.
At cost, the system was expensive. In order for Kyoto Cooling to cope with cooling only at the expense of outdoor air, it should be no hotter than 25 ° C outside. For the summer period, in addition to the “wheel”, we would still have to buy refrigerators at full capacity. In addition, the solution was new and untested for us, and we did not want to embark on experiments at such a large-scale facility.
Chiller fan coil Another alternative was the dual circuit chiller circuit on ethylene glycol and water. The external circuit with ethylene glycol antifreeze solution connects the chillers and the heat exchanger. The internal contour with water goes from the heat exchanger to cupboard conditioners. Thus, the air in the engine rooms is cooled with the help of heat carriers - water and ethylene glycol solution.
We used a similar scheme in the very first OST-1 data center. The possibility of natural cooling in the cold season is the main advantage of this scheme. With free cooling, only fans and pumps consume electricity. Compressors, the most gluttonous element of the system, are turned off.
Chillernaya scheme looked the most suitable option for the project NORD-4. It did not rest on the limitations of the freon scheme, and in the long term was more energy efficient. Unlike Kyoto Cooling, it did not require special solutions in the planning of the building, but it also supported the free cooling mode.
Chillers We analyzed the energy and economic efficiency of equipment with similar technical parameters: from the cost of equipment and its estimated energy consumption during the year it was calculated how soon the price difference compared to the cheapest model will be offset by energy savings. In fact, we compared CAPEX and OPEX. As a rule, it turned out that the more expensive the car, the longer it worked in free-cooling mode and the less it consumed electricity during the year. Thanks to this comparison, we saw that the payback period for expensive models is comparable to cheaper ones.
Viewed equipment of more than 10 vendors: Emerson, Uniflair, Stulz, Climaventa, Hiref, Aermec, Climacore, Clivet, Hitema, Clint, Ferroli. According to the results of our research, the Stulz CyberCool-2 chiller with a nominal cooling capacity of 950 kW turned out to be the most suitable for our purposes. The chiller operates on a “warm” coolant: the ethylene glycol temperature at the outlet of the chiller is 16 ° C, and the inlet temperature is 22 ° C. Typically, the temperature of ethylene glycol is +7 and +12 ° C.
The high temperature of the coolant provides a longer period of natural cooling due to outdoor air. If ethylene glycol is warmer than the outside air, the cooling system removes heat without turning on the compressors.
Energy efficiency analysis of the Stulz CyberCool-2 chiller.
The chiller switches to full free cooling mode when it is cooler on the street + 6 ° , and in partial mode - at temperatures from + 6 ° to + 19 ° . With full free-cooling, the compressors are turned off, and only fans and pumps consume electricity.
The cooling capacity of the chiller is regulated by 4-stage screw compressors. They work in four modes and can vary productivity: 0, 25, 50,100%. Power is regulated also at electronically switched fans.
The chiller controller is powered by the built-in UPS, so when switching power to the data center, it continues to work and does not waste time rebooting.
Chillers Stulz CyberCool-2.
Air conditioners. As in the case of chillers, more expensive models of air conditioners turned out to be more energy efficient. It happened with the Stulz ASD1000CW selected as a result. According to our calculations, the high cost of the air conditioner Stulz is compensated for in less than a year by the use of fans with electronic switching.
The water temperature at the inlet to the air conditioner is 18 ° C, at the outlet - 24 ° C. Due to the high temperature of the coolant in the system, air conditioners operate above the dew point, so no excess energy is spent on moisture condensation and subsequent air moistening.
Air Conditioner Stulz ASD1000CW.
Each air conditioner adjusts the cooling capacity with a 2-way valve, changing the flow rate (variable flow diagram). In a more common scheme with a constant flow rate, the air conditioner regulates the air temperature using a 3-way valve, passing part of the water through the heat exchanger and part through the bypass. At the exit of the air conditioner flows are mixed. In our scheme, this does not happen, and it turns out to keep the high temperature of the return water.
At variable flow rates, the pumps maintain constant pressure at the air conditioning inlets. This is followed by an automatic control system.
Comparison of schemes with constant and variable water flow.
In each machine room there are 14 air conditioners, in power centers - 4 pieces each. N + 1 redundancy scheme.
On the roof of the data center there are 14 chillers. The chiller groups will be installed in four stages in accordance with the launch of the machine rooms: 4 + 3 + 4 + 3. At each stage, a reserve of at least N + 1: 3 + 1, 6 + 1, 9 + 2, 12 + 2 is maintained. To date, 7 chillers (6 + 1) have been installed.
All the other elements of the system we placed in the refrigeration center on the fourth technical floor above the machine rooms. Here are the pumps, tanks, batteries, the main valves and fittings, intermediate heat exchangers, pipe wiring water and glycol circuits of the cooling system. So we simplified for ourselves the control of valves, unloaded the floors with machine rooms and protected ourselves from large-scale leaks near the IT equipment. The technical floor is insulated and equipped with 24 emergency drainage system funnels. In the event of an accident, all spilled liquid is removed from the technical floor through pipes to the basement, into special storage tanks. Leakage sensors are in the configuration of each air conditioner, there is also a network of independent sensors monitoring system.
Technical floor.
The external circuit with ethylene glycol solution connects the chiller and the heat exchanger: the warm glycol with a temperature of 22 ° C goes to the chiller, cools to 16 ° C and enters the heat exchanger. Each chiller with a pump, piping and heat exchanger forms an independent cooling unit. If one of the elements fails, the entire node is displayed.
The interaction of external and internal circuits in the chiller scheme.
The internal water circuit, which occupies the entire technical floor, connects the heat exchanger with air conditioners: water of 18 ° C flows to the air conditioners, and 24 ° C water returns from the air conditioners to the heat exchanger.
In the inner loop, battery packs of 46 cubic meters are provided. During interruptions in the operation of chillers (for example, when switching from the city to a diesel generator set ), these tanks maintain autonomous operation of the system for up to 8 minutes. It turns out a kind of "uninterrupted cooling."
Roof with chillers and technical floor.
Internal circuit pipelines on one of NORD-4 floors.
The main advantage of the resulting cooling system is its high energy efficiency. This was achieved thanks to the following solutions:
According to our calculations, with the full design load of the data center, the average annual mechanical PUE (“over the cold”) will be no more than 1.21.
The conditions of the problem are
Type of building. NORD-4 was immediately planned to be multi-storey: on the first floor of the data center there is an energy center, on the second and third - 8 rooms for 2016 counters.
Cooling capacity. To remove heat from IT equipment with a capacity of 9,000 kW and associated engineering infrastructure, at least 10,000 kW of cooling capacity was required.
Temperature conditions in the hall . In the cold corridors of the machine room, the temperature should be within 23–27 ° . In such conditions, the equipment works stably, and engineers in cold corridors do not catch cold.
')
Climatic conditions on the street. The outdoor temperature for the project is taken from building codes and regulations (SNiP). In the Moscow region, in winter the temperature drops to –42 ° , and in the warm season it reaches +37 ° .
Certification. The data center was planned to be certified according to the standards of Uptime Institute Tier III (project, finished building and operation processes). According to the standard in the Tier III data center, any component of the system can be turned off for maintenance without interruption. This means that each element has a reserve of at least N + 1.
Launch schedule. The data center was immediately built with a view to a phased launch, so all systems needed to be designed so that the installation and commissioning of new capacities did not interfere with the already operating machine halls.
Type of IT equipment. NORD-4 is a commercial data center: it will be filled gradually and with a variety of racks. This means that you can not rigidly fix the layout of the racks in the engine room and pre-isolate the hot or cold corridors.
Choice of cold supply scheme
Before we dwell on the chiller scheme, which was implemented, we considered several alternatives.
Direct expansion air conditioners (DX). This configuration is often called simply Freon. In the machine rooms are located cabinet freon air conditioners with evaporator, compressor, thermostatic valve. On the roof or near the data center building there are external blocks with a capacitor. Air conditioners and outdoor units are interconnected copper freonoprovodami.
In this scheme, the air conditioner cools the air directly in the engine room without an intermediate heat carrier (water, air).
The principle of operation of the freon cooling scheme.
This system works in 5 of our data centers, so we are well acquainted. Compared to the chiller scheme, it has a number of advantages: it is cheaper in capital investments, easier and faster to install. Since all air conditioners are autonomous, to reserve a system, it is enough to add the required number of extra air conditioners.
Despite these advantages, it was not suitable for the NORD-4 project due to the following restrictions:
- Freonline length: for effective cooling, it should not exceed 50 meters horizontally, if there is a height difference - even less. With a multi-storey data center, it was difficult to meet these limitations.
- low energy efficiency: in the long term freon scheme is less profitable than the same chiller one. The compressor freon air conditioner only two states - on and off. In this scheme, there is no possibility of cooling without a running compressor, i.e., electricity will be consumed for cooling, even when it is cold outside.
Kyoto Cooling. This is a dual circuit with natural air cooling (free cooling). Data center air circulates in the inner loop, and outside air is supplied to the outer loop. In a heat exchanger-recuperator of the rotor type, or "wheel", the hot air from the IT equipment is cooled by outdoor air. Kyoto Cooling does not involve compressors (that same natural cooling) at low street temperatures.
The principle of operation of Kyoto Cooling.
This option bribed its energy efficiency: in the free-cooling mode, electricity is spent only on the drive of the electric motors of the fans. Data centers using this scheme can boast one of the lowest PUEs - 1.09–1.13. Unlike the chiller circuit, Kyoto Cooling works without water, so there is no risk of leakage.
At the same time, Kyoto Cooling has specific requirements for building design:
- the heat exchanger "wheel" with all ventilation chambers takes up a lot of space. In fact, it needs a separate technical room, which is difficult to fit into the finished building project.
- air has a small heat capacity: large air ducts are needed to transfer a large amount of heat. The function of air ducts can be performed by corridors, but this, again, must be foreseen at the design stage of a building.
- inside the machine room it is necessary to isolate the hot corridors. This is difficult to do when racks of different sizes and the room is filled gradually.
- additional costs for filters. The outside air that comes to the wheel must be filtered, otherwise the dust will get into the engine rooms.
"Wheel" with a horizontal axis.
At cost, the system was expensive. In order for Kyoto Cooling to cope with cooling only at the expense of outdoor air, it should be no hotter than 25 ° C outside. For the summer period, in addition to the “wheel”, we would still have to buy refrigerators at full capacity. In addition, the solution was new and untested for us, and we did not want to embark on experiments at such a large-scale facility.
Chiller fan coil Another alternative was the dual circuit chiller circuit on ethylene glycol and water. The external circuit with ethylene glycol antifreeze solution connects the chillers and the heat exchanger. The internal contour with water goes from the heat exchanger to cupboard conditioners. Thus, the air in the engine rooms is cooled with the help of heat carriers - water and ethylene glycol solution.
We used a similar scheme in the very first OST-1 data center. The possibility of natural cooling in the cold season is the main advantage of this scheme. With free cooling, only fans and pumps consume electricity. Compressors, the most gluttonous element of the system, are turned off.
Chillernaya scheme looked the most suitable option for the project NORD-4. It did not rest on the limitations of the freon scheme, and in the long term was more energy efficient. Unlike Kyoto Cooling, it did not require special solutions in the planning of the building, but it also supported the free cooling mode.
Equipment selection
Chillers We analyzed the energy and economic efficiency of equipment with similar technical parameters: from the cost of equipment and its estimated energy consumption during the year it was calculated how soon the price difference compared to the cheapest model will be offset by energy savings. In fact, we compared CAPEX and OPEX. As a rule, it turned out that the more expensive the car, the longer it worked in free-cooling mode and the less it consumed electricity during the year. Thanks to this comparison, we saw that the payback period for expensive models is comparable to cheaper ones.
Viewed equipment of more than 10 vendors: Emerson, Uniflair, Stulz, Climaventa, Hiref, Aermec, Climacore, Clivet, Hitema, Clint, Ferroli. According to the results of our research, the Stulz CyberCool-2 chiller with a nominal cooling capacity of 950 kW turned out to be the most suitable for our purposes. The chiller operates on a “warm” coolant: the ethylene glycol temperature at the outlet of the chiller is 16 ° C, and the inlet temperature is 22 ° C. Typically, the temperature of ethylene glycol is +7 and +12 ° C.
The high temperature of the coolant provides a longer period of natural cooling due to outdoor air. If ethylene glycol is warmer than the outside air, the cooling system removes heat without turning on the compressors.
Energy efficiency analysis of the Stulz CyberCool-2 chiller.
The chiller switches to full free cooling mode when it is cooler on the street + 6 ° , and in partial mode - at temperatures from + 6 ° to + 19 ° . With full free-cooling, the compressors are turned off, and only fans and pumps consume electricity.
The cooling capacity of the chiller is regulated by 4-stage screw compressors. They work in four modes and can vary productivity: 0, 25, 50,100%. Power is regulated also at electronically switched fans.
The chiller controller is powered by the built-in UPS, so when switching power to the data center, it continues to work and does not waste time rebooting.
Chillers Stulz CyberCool-2.
Air conditioners. As in the case of chillers, more expensive models of air conditioners turned out to be more energy efficient. It happened with the Stulz ASD1000CW selected as a result. According to our calculations, the high cost of the air conditioner Stulz is compensated for in less than a year by the use of fans with electronic switching.
The water temperature at the inlet to the air conditioner is 18 ° C, at the outlet - 24 ° C. Due to the high temperature of the coolant in the system, air conditioners operate above the dew point, so no excess energy is spent on moisture condensation and subsequent air moistening.
Air Conditioner Stulz ASD1000CW.
Each air conditioner adjusts the cooling capacity with a 2-way valve, changing the flow rate (variable flow diagram). In a more common scheme with a constant flow rate, the air conditioner regulates the air temperature using a 3-way valve, passing part of the water through the heat exchanger and part through the bypass. At the exit of the air conditioner flows are mixed. In our scheme, this does not happen, and it turns out to keep the high temperature of the return water.
At variable flow rates, the pumps maintain constant pressure at the air conditioning inlets. This is followed by an automatic control system.
Comparison of schemes with constant and variable water flow.
Final implementation
In each machine room there are 14 air conditioners, in power centers - 4 pieces each. N + 1 redundancy scheme.
On the roof of the data center there are 14 chillers. The chiller groups will be installed in four stages in accordance with the launch of the machine rooms: 4 + 3 + 4 + 3. At each stage, a reserve of at least N + 1: 3 + 1, 6 + 1, 9 + 2, 12 + 2 is maintained. To date, 7 chillers (6 + 1) have been installed.
All the other elements of the system we placed in the refrigeration center on the fourth technical floor above the machine rooms. Here are the pumps, tanks, batteries, the main valves and fittings, intermediate heat exchangers, pipe wiring water and glycol circuits of the cooling system. So we simplified for ourselves the control of valves, unloaded the floors with machine rooms and protected ourselves from large-scale leaks near the IT equipment. The technical floor is insulated and equipped with 24 emergency drainage system funnels. In the event of an accident, all spilled liquid is removed from the technical floor through pipes to the basement, into special storage tanks. Leakage sensors are in the configuration of each air conditioner, there is also a network of independent sensors monitoring system.
Technical floor.
The external circuit with ethylene glycol solution connects the chiller and the heat exchanger: the warm glycol with a temperature of 22 ° C goes to the chiller, cools to 16 ° C and enters the heat exchanger. Each chiller with a pump, piping and heat exchanger forms an independent cooling unit. If one of the elements fails, the entire node is displayed.
The interaction of external and internal circuits in the chiller scheme.
The internal water circuit, which occupies the entire technical floor, connects the heat exchanger with air conditioners: water of 18 ° C flows to the air conditioners, and 24 ° C water returns from the air conditioners to the heat exchanger.
In the inner loop, battery packs of 46 cubic meters are provided. During interruptions in the operation of chillers (for example, when switching from the city to a diesel generator set ), these tanks maintain autonomous operation of the system for up to 8 minutes. It turns out a kind of "uninterrupted cooling."
Roof with chillers and technical floor.
Internal circuit pipelines on one of NORD-4 floors.
findings
The main advantage of the resulting cooling system is its high energy efficiency. This was achieved thanks to the following solutions:
- The increased temperature of heat carriers - ethylene glycol and water.
- Variable coolant flow in the internal water circuit. This scheme eliminates the mixture of hot and cold water. As a result, we received an increased temperature difference between hot and cold water and a longer period of free cooling.
- Fans with electronic switching in cabinet air conditioners and chillers.
According to our calculations, with the full design load of the data center, the average annual mechanical PUE (“over the cold”) will be no more than 1.21.
Source: https://habr.com/ru/post/314730/
All Articles