A brief excursion into the data center cooling system. Advantages and disadvantages

Somehow it happened that Habré announced the day of the data center cooling.



It is necessary to participate, because In this topic, I soon two decades.

KDPV - small.





')

Excursion - short. Very brief.



Freon



Good old. He's a DX. It is a direct extension. He is Gogh ...



Pros:

• Cheap and angry;
• Simply organize any degree of reliability to obtain a certificate from the Uptime Institute;
• It is difficult in the world to find a place where there is no specialist who could serve it and / or repair it.

Minuses:

• The highest PUE (which is bad. I will not explain here now about PUE. It’s necessary to write a separate article about it. Everything is not as simple as it might seem at first glance. But another time.);
• Any power failure and ... no cooling. Those. or put the UPS on all freon air conditioners, or find a balance between the load and the volume of air in the machine room, so that cold air is enough for the time of the UPS operation; (here, by the way, lies the main argument against the apologists of the “cold corridors”. But about the “corridors” is also another sad story.) There is also a problem with the UPS for freon air conditioners - the compressor starts up “eats” from 3 to 7 denominations. Typical UPS with such a mockery - throws the load. There are, of course, air conditioners with a frequency regulator for the compressor, but ... so far these options do not work at all with low-temperature kits in winter. Not ready to explain the physics of the process, but from the experience of respected people - this is so. As a result, either a UPS with a sevenfold margin, or a reload on the volume of air in the room;
• It works badly at too low loads. Either it is necessary to abandon the "precision", or even put heaters in the room in order for the load on the air conditioner to fall within the allowable range, or to put up with frequent replacement of compressors. What is worse - many air conditioners have non-switchable protection against too frequent switching on of compressors and the air conditioner simply turns off with an error message. (This is another separate topic, about the features of the freon cycle ...);
• There are risks with dramatic significant changes in the load, as, for example, in supercomputers. For example: they considered the task, the load dropped from 100% to 5%, a new task was loaded, the load went up from 5% to 100 ... But in less than 3 minutes! And some air conditioners have a timer that blocks the compressor on for 3 minutes after it is turned off ... And at 100% load in the supercomputer in the machine room, the temperature increase is from degrees per second and higher ... Then, as lucky;
• Big problems with work at very negative temperatures. -10 - -15 nothing yet. -30 and below - pulls additional receivers, valves and automatics. Not everyone can design. Not everyone can customize. Not all "low-temperature" kits actually withstand the declared temperatures. If you need a Freon that will work exactly at -40, contact companies that have living objects in regions where it is -40 in winter.
• Slightly less scary in fresh models of air conditioners. But still - lower humidity. The typical evaporator temperature of a freon air conditioner is below the dew point for standard 20C 50% RH for the turbine hall. Accordingly, air humidification is required. Depending on the method of implementation of humidification, the design of the room, ventilation settings, it can pull another 0.1-0.2 in plus to the PUE ...

Freon with free cooling



The same DX, but in the air conditioner cabinet there is another heat exchanger in which the glycol solution (ethylene) lives. At temperatures below about five degrees from the "set point" - the freon circuit is turned off, the pump in the "glycol circuit" is turned on, and all the costs of cooling are electricity for the operation of the pump and fans.



Pros:

• All the same as for freon, because if there is a problem with the glycol circuit, the freon part of the air conditioner will automatically return to about work;
• A noticeable decrease (which is good) PUE even in hot countries;
• With proper redundancy in the glycol part, the use of low-temperature kits in the freon part is not required. Trifle, but in Norilsk, for example, it may be useful.

Minuses:

• If we get lucky. Depending on the firmware, settings, manufacturer, you can get all the drawbacks of the Freon system in the summer, and in the winter get all the drawbacks of the chiller system with free cooling (it will be described a little later). At least - in the summer there will be all the problems with ensuring uninterrupted power;
• Complication of the internal block of the system with a corresponding decrease in the number of craftsmen who will undertake to repair it;
• Doubling of pipelines (not quite, but still ...);
• Doubling the space for external blocks (almost always), since glycolic dryclayers are added to freon circuit capacitors;
• As a result of all the above minuses - the system is more expensive in design, construction and operation.

Chiller system



Here you can write a series of articles for a year of daily publications. Well, or a very thick tutorial.

There are lots of options for implementation.



Short:



Chiller is a device for cooling a fluid and maintaining a predetermined (low) fluid temperature at the outlet.



As a result, the cooling system consists of chillers that cool water, pipelines with fittings and very simple air conditioners (you can often find the word “closer” in Russian-speaking sources) containing a water-air heat exchanger, a control valve and a fan.



Globally, what is described in the pros and cons in its pure form is almost never used.



Pros:

• Due to the greater heat capacity of water, about the complete absence of problems with highly variable loads. Especially if the following plus is applied;
• You can put the tank in the circuit. Big. Or very big. Almost free on the background of the price of chillers. And pour the cooled water first there. Then, in case of any problems in the cooling system or power supply, it is enough to circulate the fluid in the circuit so that computers do not overheat immediately, in 15-40 seconds, as for a regular supercomputer, for example, and overheat in 15-20 minutes for which even a sleeping admin already will wake up and either fix everything or extinguish everything.
• To ensure uninterrupted operation, it is sufficient to power the circulating pumps from the UPS. If pumps with a frequency regulator, then do not even need to give a margin for starting currents;
• Yes, you need more specialists for maintenance, you need a plumber in addition to a freon air conditioner, but it’s hard to find a place on the planet where there is not at least one qualified plumber.

Minuses:

• Very much depend on how the system is arranged. Too many implementation options;
• In general, it is much more expensive than the freon system;
• In large systems, you have to make two circuits - glycolic "on the street" and through the heat exchanger "water" inside the data center / building;
• PUE is the same or worse than the "Freon". Since circulation pumps are added to the freon.

Chillernaya system with frikulingom



Option 1. Which is not so good.



This is a chiller system. But it has either a separate group of drycoolers, or drycoolers built into the general construc- tion, to which the system switches at the moment when all the necessary cooling capacity can be provided by free-cooling.



Pros:

• “Half of the year is bad weather, Half of the year is absolutely nowhere” ... While the weather is bad, it is cold and disgusting outside ... cooling is almost free. Only pumps and fans work;
• Simple implementation for most variations of chiller systems;
• North of Sochi - quite a good PUE.

Minuses:

• Only that there is a more correct option, if by and large.

Chillernaya system with the right frikulingom



Until relatively recently, there was only one manufacturer. Roles now patents expired, we bought all licenses, but there were more options.



Pros:

• All that and for the chilling system with free cooling;
• In addition, if the air temperature outside is below the coolant temperature at the entrance to the chiller, the liquid first goes to counter free cooling. If the liquid still has a temperature above the setpoint after the free cooling circuit, the liquid is sent to the freon evaporator and the freon circuit is connected to the operation. It seems to be a trifle, but for Moscow it increases the average freezing time by 2-4 months for an average year and, sometimes, saves electricity even in July.

Minuses:

• Yes, in general, they are not. (Did you notice? This is my favorite option!). Yes, the system turns out to be more expensive than freon, and slightly more expensive than the usual chiller system. But, oddly enough, it is usually no more expensive than a “regular” chiller system with free cooling.

Air freaking



There are roughly three options - a rotary heat exchanger, an open air circuit and a closed air circuit. Plus, in each of the options, you can add "adiabatics" ...



Rotary heat exchanger



There is an internal air circulation circuit, where the heated air enters the installation, passes through half of the slowly rotating “wheel”, where it cools and goes to the workshop. And the outer contour. Where the outside air enters the installation, passes through the other half of the slowly rotating “wheel”, where it heats up, and goes outside. Plus, usually, in case of severely hot weather, an evaporator is added to the system at the outlet of the inner loop and a condenser at the exit of the outer loop.



Pros:

• Good pue;
• Approximately closed internal circuit, air humidification is rarely required;
• Freon circuit is used only in very hot weather.

Minuses:

• The impossibility of complete tightness of the joint with the heat exchange wheel. As a result, there are overflows of air;
• Because of this, there are potentially problems with fire detectors when something is burning outside;
• For some reason, manufacturers want a lot of money for this system. Despite the seeming simplicity.

Open air circuit



Facebook is the most famous, but there are still several manufacturers and installations. In general, this option can be approximately always driven into the price-quality framework set by the customer.



Pros:

• A very simple system in terms of mechanics - fans, two controlled flaps;
• Cheap at construction "in the open field" (Green Field);
• Very good pue. Costs only for fans, humidification and filters;
• Well implemented for modular data centers and container mining farms.

Minuses:

• Rain. Yes, he is the most: wet nasty drops of water on top. Once a cloud gathered from a ceiling in the data center from which it started to rain ... And just a small bug in the temperature and humidity control system ...;
• The presence of a freon cooling system is required if the temperature “overboard” can be higher than the allowable temperature in the machine room;
• It is required to have 100% power on freon air conditioners if you need to continue working in case of fire outside the data center (for example, for Moscow and the surrounding area, in case of burning peatlands);
• Humidification in low humidity areas may require large amounts of water.

Closed air circuit



Usually using an air-to-air cross-flow heat exchanger. The internal circuit air is cooled by external circuit air. Without adiabatics, there are approximately the same pros and cons as for the rotary heat exchanger and open circuit, except for:



Pros:

• No air flow;
• Reduced humidification costs compared to open contour and wheel;
• It is not required to change the mode of operation in case of fires outside.

Minuses:

• The temperature inside will always be higher than the outside temperature, if you do not use the freon system for additional cooling during the warm season. Or, plus, adiabatics ...

Freekuling with adiabatic (I also like this option very much ...).



The general sense is that the heat of evaporation of water is approximately 2.2 MJ per kg. Water in the air stream will almost always evaporate. If we spray water in the air stream, we will not only increase the humidity, but also lower the temperature.



Pros:

1. When properly implemented (with a closed loop), 15-20% of the power of the freon system of the total capacity of the cooling system is sufficient for operation in most parts of Europe;
2. If implemented correctly, PUE can be 1.1 for most of Europe and lower for most of Russia.

Minuses:

• Quite a big consumption of water. Starting at temperatures of the order of +12 - +14 degrees;
• Tier III requires either a backup water source or water supply for the entire required operating time from a diesel generator set (approximately 36 hours ... This is a lot of water, reserved pumps for its pumping, three taps for each tee and other joys of Tier III);
• In some regions, water is expensive. Very expensive. And even not so much “expensive” as “great value” and the locals will simply arrange for you a little pogrom if they know that you do not drink this water ...

Direct "water" cooling



It is either not entirely direct, or not entirely water. But…



Pros:

• There is no noise in the workshop;
• ... is there anything else?

Minuses:

• A huge number of joints, joints, tubules, clamps, etc. As long as you need to cool two or four processors and related chipsets, everything is great. When it comes to 15,000 dual-processor servers in the cloud data center ... Suddenly it turns out that the system is flowing. Is always. Because the joints in it are hundreds of thousands. Have you ever seen a plumber in an apartment building banging a year in a utility room without ever having a call to fix a leak? There is no such thing. And in an apartment building, compared to a small water-cooled data center, there are hundreds of times fewer joints and mates in which leakage can occur;
• Water. Any leakage, leakage, spillage - this is at least loss of the work of one rack. If you're lucky. As the practice of data centers using such cooling shows, this is typically more than a day of downtime of the entire infrastructure of a heap of data centers due to one of the important switches flooded with water.
• Great difficulty with materials. If there is aluminum in contact with water in the system, there should not be anything containing copper in the system. Up to the point that you need to carefully select stainless steel alloys;
• Which leads to the next group of problems. To make all radiators only from copper is very expensive. And very hard. Stupidly, the weight of the rack is such that it is easier, instead of reinforcing the floors, to fill all the basements and floors under the main hall with concrete. So there will be aluminum. So, you need to carefully select all materials that come into contact with water. Otherwise - all pipes, channels, radiators will be filled with white porridge. Which, when used correctly in an instrument, is called “corundum” ... And this also means that the impellers that were turned off at all pumps ...;
• BRS (quick detachable connections). Good ones are more expensive than the entire rack, along with platforms, processors, memory, disks. And they need at least 80 on the rack ... All the rest - cause the very downtime for several days. And this is not to mention the trifle that "those" "good" BRS are not enough. For boosters and some military aircraft ... I tried the BRS from excavators. Does not fit. (but this is another story again ...)
• In any case, just water can not be used. This or some solution of water, glycol and corrosion inhibitors, or distilled water, or some other liquid. It costs money. Even banal distilled water - from 10 rubles per liter. Or water treatment system. Which ... (you will not believe!) ... also worth the money!

Immersion cooling systems



There are also many options. Consider the three most fashionable and frequent.



Submersible cooling using mineral oil.



Pros:

• … Interest Ask. Are they there? Please tell me if there is. (and yes, I'm not asking about the scale of 2-3 units, but about the scale of 8000 and more units ...)

Minuses:

• For me personally, the worst. Phobia, or what? I hate it when my hands are slim and in oil ... If it were not for this, I would have been happily turning the nuts in some service station ...;
• Regardless of phobias - all in oil. Oilproof overalls from shoes to cap for everyone who comes to the workshop;
• Compatible with plastics. There is never any guarantee that the case of any capacitor / resistor / processor will not melt in mineral oil. And this is not a warranty case;
• High viscosity. Consequently, high energy consumption for circulation in the system;
• High viscosity. Therefore, the probability that some hot element on the board will stagnate hot slurry. And yes, people did come across this, but, excuse me, the details are “classified”;
• In any case, you need a heat exchanger and an "external circuit" with a glycol solution. Otherwise, there are too many problems in winter;
• As a result, the price tag is not much different from the chiller system. With a bunch of problems and employees who refuse to go to the gym. Well, yes, the organization that is trying to sell this thing ... They have all the same renders on the site that were hanging 10 years ago. Not a single real installation ...
• For those who are going to formally design such a system and pass the examination - fire load. Oil - from the point of view of the law - a flammable liquid. Further ambushes begin - Any volumes greater than about 1000 liters - just like that can not be placed in the room. And 1000 liters is, sorry, one bath for 24 units plus strapping. Anything more is the easiest thing to do through the development of STU. But this is in any case - other requirements for adjacent rooms, other requirements for the presence of vestibules, other requirements for distances to emergency exits. On average - if you put a container in an open field, in kilometers from residential buildings - there is a chance to formalize everything for more or less reasonable money. If you are trying to deliver this in Moscow, St. Petersburg, Yekaterinburg or Novosibirsk - do not forget to have on hand a supply of crackers in case of a sudden visit from the inspection bodies.

Immersion cooling with novek 7000 series



In this series of liquids, the NOVECK boiling point is greater than 60C. Accordingly, the bath with Novak, heat exchanger with "water". Something to mix or just convection.



Pros:

• Works;
• You can make a Tier III, although it is not very trivial then to prove to Aptaym that it is a Tier III;
• Novec on the foreseeable perspectives such as the big decades does not lead to damage to the microchip case, oxidation of contacts, etc .;
• It is still considered harmless to homo-sapiens, although competing organizations are trying to tell stories about liver damage ...
• You can try to do without taking into account the plastic and server boards in the calculation of the fire load. It seems a trifle, but sometimes it is the difference between ordinary lamps and an order of magnitude more expensive explosion-proof lamps.

Minuses:

• Price. Actually, I haven't met any other cons yet. But if you start counting payback ... Not very fast.

Immersion cooling with boil / condensation novech.



Pros:

• The coolest topic;
• It is impossible for the components to overheat until the whole Novec boils away;
• The smaller amount of Novak compared to immersion cooling without boiling / condensation is potentially thousands of dollars for a “bath” of cost difference.

Minuses:

• Some other novech. Did not find his numbers. There should be a boiling point of 40-50 degrees.
• A part of Novak will evaporate and fall outside the bath with heat exchangers. Have to buy more, to top up from time to time;
• Price tag About the same hundred cu per kg The same density in the region of one and a half kg per liter.
• To drive slush for $ 100 per liter to the street - too expensive. So, there will be heat exchangers, drycoolers, glycol, rezivirovanny pumps ...
• Automation, which nobody has done for real data centers yet ... Trifle, but not all owners of data centers want to be the first observer of rain from Novek. Although it is rain, formally, and absolutely safe for electronics ...

Findings:



There are already a lot of cooling system options. Before you make the final decision on the choice of options for your data center - you need to carefully consider the money. How much money you are ready to invest right away, how much money you are willing to spend on the fact that your data center will be “green”, how quickly you want to “discourage” your investments on “greenness”.