Novec-based immersion cooling server or farm as an alternative to air cooling
Hello!
We continue to talk about our products in the last article, Novec 1230 Dry Water to protect the server and not only there were many questions about innovations in the field of server cooling, so we decided to highlight the answers to many questions in a separate post! In addition, most recently our solution was used on the largest crypto farm in Hong Kong !
Learn the formula from the board?
')
The article discusses the features and benefits of a passive two-phase immersion cooling system for servers based on fluoroketones . In the article you are waiting forintrigue and investigation, analysis of technology and experiments!
We first discuss the limitations of the traditional air cooling system. The reasons for the low efficiency of the traditional air cooling system are: the effect of the second law of thermodynamics (irreversibility of thermal processes) due to a variety of heat transfer processes, mixing of cold and heated air flows, high power consumption of cooling equipment - chillers, air conditioners, etc., as well as the use air as a heat transfer link. When introducing these technologies, they are currently taking into account that their effectiveness has been reduced for one or several of the above reasons.
Water-cooled rear doors, duct air cooling system, forced circulation stands restrict air flow mixing. These and some other technologies allow you to operate the system without a chiller, switching to the use of an economizer when the weather permits.
Systems with a constantly working economizer are simpler in their device and can achieve an energy efficiency indicator of <1.3. Such server should be located in areas with relatively cold climates.
It is also worth taking into account other inherent economic aspects and environmental impact. The ability to control airflow at the chassis, rack, or server level adds significant cost when installing each new or expanding existing data center.
Therefore, the question of increasing energy efficiency should be considered not only from the point of view of how to remove excess heat, but also from the point of view of how to use it. However, the possibility and cost of recovering the heat removed at any distance from the data center is limited by a large volumetric air flow and a low value of its useful work.
Liquid cooling can reduce the effects of the aforementioned reasons for the low efficiency of air cooling, facilitate the recovery of the heat released and increase its thermodynamic availability. In one study, a comparison was made between two cooling systems for a supercomputer: a hybrid air / liquid cooling system and an air system. In the same study, the efficiency of a completely liquid cooling system and the same system operating in the absence of chillers or with a water economizer was predicted.
The latter configuration allows you to save up to 90% of the energy for cooling as compared to an air-cooled cluster.
However, the introduction of traditional liquid cooling systems, whether single- or two-phase, closed or immersed, is complicated by the number and variation of heat-producing devices in the server and the requirement that a hot-swap procedure can be performed for each server within the rack. ) . This complicates the task of directing all the heat produced on the circuit board to the external flow of coolant.
As a result, the hybrid air-liquid cooling system introduces the cost of design and production of cooling plates, backup pumps, water supply, quick-disconnect couplings, sensors and heat exchangers.
A completely liquid cooling system is often even more complicated and requires the installation of additional cooling plates, grabs (lifts with a clinging device) and sealed electrical connectors. The effectiveness of many of these systems is limited by the secondary or even tertiary thermal interface and the temperature slip of the coolant (this phenomenon is a change in the temperature of the boiling liquid as a result of a change in the composition) . Also in systems where working fluids are hydrofluorocarbons or perfluorocarbons, leakage may occur, leading to the release of substances with high global warming potential into the atmosphere.
Thus, a simple, compact cooling system is needed that minimizes the use of natural resources and emissions of harmful substances. In this case, all the heat produced should be removed while minimizing the temperature difference between the active layer in the processor chip and water as the secondary coolant. The system should be modular, scalable with easy customization for new equipment.
Passive two-phase immersion cooling system has long been used to cool expensive electronic equipment such as transformers, traction converters, special-purpose computers and klystrons. This technology is relatively simple, reliable and effective.
Traction dump truck dump truck.
These systems typically use pressure vessels and sealed electrical connectors. Tanks vacuum and fill almost as well as refrigerators and their maintenance in the field is impossible. Creating a similar cooling system for computers will be expensive and difficult due to the huge number of replaceable components and connectors. For this reason, many people do not even consider immersion cooling in the context of datacom equipment.
These devices are widely used in the world for precision cleaning of various parts, ranging from screws and bearings to orthopedic implants, printed circuit boards and injection nozzles.
Installation of steam degreasing company Reibesam.
The following article will be devoted to this technology.
The degreaser is an open rectangular tank with two rows of cooling coils installed on top of the perimeter. Up to a certain height, the tank is divided into two parts, or baths filled with a volatile solvent. In the first bath, the solvent is heated from the bottom and boils. Couples rise to the height of the first row of cooling coils, creating a saturated steam zone below this level. The condensate flows downwards and flows through the water separator into the rinse bath. Thus, as a result of distillation, only the solvent free from contamination enters this bath.
Scheme of the device steam degreaser.
These systems are capable of cleaning thousands of parts per shift, consuming a small amount of solvent. At the same time, most of the time they are either fully open or covered with a horizontally moving lid when not in use. Minimization of losses of solvent as a result of ablation is carried out at the expense of secondary cooling coils operating at temperatures below 0 ° C.
This concept is based on the premise that electronic equipment can be cooled by immersion in a half-open bath, which is in many ways similar to the steam degreaser discussed above. The term “half-open” means that the bath is closed when access to equipment is not required and, as an example of such a design, a chest freezer for food can be given. Just like him, the bath operates at atmospheric pressure and does not have special hermetic connectors for supplied and discharged electricity.
In this system, each server or node is connected to the backplane at the bottom of the tub (as opposed to the back of the server rack). The bath is partially filled with volatile dielectric fluid.
Model open submersible cooling system.
Electrical wiring below the level of the liquid is brought into the channel, and out of it at the top of the tank. The built-in vapor condenser is cooled with either tap water or water used for comfortable heating (yes, there is such a concept in the USA) .
In addition, vapors can passively flow into the so-called natural draft cooling tower, giving off heat directly to the outside air, without using water as an intermediate coolant.
The Allied Control project with 3M fluoroketone is a two-phase immersion cooling in an open bath.
The concept has many advantages over traditional liquid cooling systems. The basic principle is that there is no need for most of the equipment that is mandatory for air and liquid cooling, and, accordingly, issues related to its installation, reliability and power consumption are not considered. The power density and reliability of the system is very high. Moreover, this technology is a priori inherent fire-resistant properties.
Of course, within this model, other aspects should be considered, for example, loss of working fluid as a result of ablation. But, since they occur in one particular place, and not in an innumerable number of connectors, they can easily be counted and reduced with the help of techniques that we will discuss later.
Inside the baths, heat from dozens of servers causes Novec fluid to boil.
Her vapors are condensed on a cooling coil and returned to the system.
The thermal efficiency of the system consists of two components. The first is determined by the design of the printed circuit board and is quantitatively expressed in the temperature difference between the active layer in the processor chip, as the main element that needs to be cooled, and the temperature of the working fluid. The second component is determined by the temperature difference between the working fluid and the supplied water. In this case, the temperature of the working fluid, we mean its boiling point at atmospheric pressure.
The processor in its typical configuration in the form of a chip, a substrate and a heat-spreading lid with a built-in radiator is almost ideal for passive immersion cooling systems. In most cases, only a 100 micron layer of a porous metal coating is required, which improves heat transfer during boiling. These coatings provide a heat transfer coefficient of> 10 W / cm2 * K with a heat flux of 30 W / cm2.
If the coating operation is introduced already at the stage of the processor production, then there will be no need for a secondary thermal interface, commonly used in many liquid cooling schemes.
Model submersible 2-phase cooling system 3M.
The next point to consider is an assessment of how much power can be cooled by this system. In one study, the authors suggest that 100 cm3 of working fluid is required to cool a 1 kW module, provided that its configuration is known and taken into account, that is, the density of the components.
For the experiment, a model printed circuit board was assembled with 20 heat-generating ceramic elements with dimensions of 19 * 19 mm and a power of 200 W each. At the same time, on the one hand, a copper radiator of 30 * 30 * 3 mm is attached to them with epoxy glue, with the coating applied on the opposite side, increasing the heat transfer during boiling. Thermocouples in the working fluid and in each of the elements allow us to calculate the individual thermal resistance and make sure that the elements do not go into film boiling mode. Next, this model PCB is immersed in a narrow vertical container of the same shape with gaps of 4 and 7 mm between the boiling surface and the wall.
Boil Novec on a crypto farm.
During the experiment, it was shown that this configuration is capable of removing 4 kW of heat (200 W from each element) through a 4 mm gap at atmospheric pressure if Hydrofluoroether is used as the working fluid — C3F7OCH3. At the same time, 4 kW is equivalent to a heat flux of 11.7 W / cm2 against a flux of 1.7 W / cm2 observed in the Cray X1E supercomputer when cooled by spraying!
The results of the experiment suggest that a value of 1 kW of heat output per 100 cm3 of working fluid is definitely attainable. Also significantly reduced the amount of materials used and various kinds of emissions.
The table below presents the properties of one hydrofluorine ester and two fluoroketones. They have the required thermal characteristics, are safe and compatible with various materials and have been tested in an open submersible cooling system.
Note that the dielectric characteristics of fluoroketones are similar to those of perfluorocarbon C6F14, which was often used in immersion cooling systems. At the same time, hydrofluoroether has a higher dielectric constant and lower resistance, which may limit its use in some cases. The first of the presented fluoroketones with a global warming potential value of only 1, is now widely used in the world as a fire extinguishing agent.
Indeed, this is the main factor affecting the viability of the presented system.
There are formulas for calculating fluid losses during filling, start-up and operation. They make it possible to conclude that the most effective measure to reduce losses as a result of the entrainment of vapors will be the creation of such a structure in which the ascending vapors will be captured using a secondary cooling coil, which automatically turns on when the system temperature exceeds the specified allowable temperature.
These racks contain mining ASIC boards immersed in Novec fluids (located in Hong Kong).
In the course of the experiments described above, it was shown that when using commercially available fluoroketone with a boiling point of 49 ° C in an open bath and a flow of water of 15 gallons (which is approximately equal to 3.785 * 15 = 56.8 liters) per minute, the temperature in the active layer of the chip processor will not exceed 60 ° C. It is enough to use water with a temperature of 28 ° C.
If it is acceptable to raise the temperature in the active layer of the processor to 83 ° C, and the volume flow of water will reach 30 gallons per minute, water can be used for primary coils with a temperature of 62 ° C.
The power density that the capacity is capable of cooling is 130 kW / m2, which is well above the 52 kW / m2 limit that is typical for racks with air or hybrid cooling. In the transition to a server scale with a completely liquid cooling system of this type, we get 25 kW / m2 against 2.2 kW / m2 for an air cooled server.
In addition, this system saves space, as it does not require the installation of additional equipment, mandatory for an air cooling system. Exhaust heat can be used for heating buildings, heating greenhouses and other objects.
An example of the implementation of the new cooling concept can be found here .
On our site .
Call us at the multichannel phone +7 495 784 7474.
Ask questions in the comments, we will try to answer all!
Of course!
" Dry Novec 1230 water to protect the server and not only
» How to safely read Habr at work using our information security screens
" From spray to film in one zilch. Paint Defender: protective film on the car body
» Tracing and e-marking technology for underground utilities. Part 1
If we made any mistakes, write to the LAN, we will fix everything promptly. Do not forget, we will correct the error, and your comment will remain hanging.
Thank you for your attention, we hope this article has been helpful to you!
UPD: Thanks for pointing out the WarP Bitcoin farm.
We continue to talk about our products in the last article, Novec 1230 Dry Water to protect the server and not only there were many questions about innovations in the field of server cooling, so we decided to highlight the answers to many questions in a separate post! In addition, most recently our solution was used on the largest crypto farm in Hong Kong !
Learn the formula from the board?
')
The article discusses the features and benefits of a passive two-phase immersion cooling system for servers based on fluoroketones . In the article you are waiting for
Air cooling system
We first discuss the limitations of the traditional air cooling system. The reasons for the low efficiency of the traditional air cooling system are: the effect of the second law of thermodynamics (irreversibility of thermal processes) due to a variety of heat transfer processes, mixing of cold and heated air flows, high power consumption of cooling equipment - chillers, air conditioners, etc., as well as the use air as a heat transfer link. When introducing these technologies, they are currently taking into account that their effectiveness has been reduced for one or several of the above reasons.
Water-cooled rear doors, duct air cooling system, forced circulation stands restrict air flow mixing. These and some other technologies allow you to operate the system without a chiller, switching to the use of an economizer when the weather permits.
Systems with a constantly working economizer are simpler in their device and can achieve an energy efficiency indicator of <1.3. Such server should be located in areas with relatively cold climates.
It is also worth taking into account other inherent economic aspects and environmental impact. The ability to control airflow at the chassis, rack, or server level adds significant cost when installing each new or expanding existing data center.
Therefore, the question of increasing energy efficiency should be considered not only from the point of view of how to remove excess heat, but also from the point of view of how to use it. However, the possibility and cost of recovering the heat removed at any distance from the data center is limited by a large volumetric air flow and a low value of its useful work.
Consider traditional liquid cooling systems and their limitations.
Liquid cooling can reduce the effects of the aforementioned reasons for the low efficiency of air cooling, facilitate the recovery of the heat released and increase its thermodynamic availability. In one study, a comparison was made between two cooling systems for a supercomputer: a hybrid air / liquid cooling system and an air system. In the same study, the efficiency of a completely liquid cooling system and the same system operating in the absence of chillers or with a water economizer was predicted.
The latter configuration allows you to save up to 90% of the energy for cooling as compared to an air-cooled cluster.
So what's the problem?
However, the introduction of traditional liquid cooling systems, whether single- or two-phase, closed or immersed, is complicated by the number and variation of heat-producing devices in the server and the requirement that a hot-swap procedure can be performed for each server within the rack. ) . This complicates the task of directing all the heat produced on the circuit board to the external flow of coolant.
As a result, the hybrid air-liquid cooling system introduces the cost of design and production of cooling plates, backup pumps, water supply, quick-disconnect couplings, sensors and heat exchangers.
A completely liquid cooling system is often even more complicated and requires the installation of additional cooling plates, grabs (lifts with a clinging device) and sealed electrical connectors. The effectiveness of many of these systems is limited by the secondary or even tertiary thermal interface and the temperature slip of the coolant (this phenomenon is a change in the temperature of the boiling liquid as a result of a change in the composition) . Also in systems where working fluids are hydrofluorocarbons or perfluorocarbons, leakage may occur, leading to the release of substances with high global warming potential into the atmosphere.
Thus, a simple, compact cooling system is needed that minimizes the use of natural resources and emissions of harmful substances. In this case, all the heat produced should be removed while minimizing the temperature difference between the active layer in the processor chip and water as the secondary coolant. The system should be modular, scalable with easy customization for new equipment.
Existing liquid cooling systems
Passive two-phase immersion cooling system has long been used to cool expensive electronic equipment such as transformers, traction converters, special-purpose computers and klystrons. This technology is relatively simple, reliable and effective.
Traction dump truck dump truck.
These systems typically use pressure vessels and sealed electrical connectors. Tanks vacuum and fill almost as well as refrigerators and their maintenance in the field is impossible. Creating a similar cooling system for computers will be expensive and difficult due to the huge number of replaceable components and connectors. For this reason, many people do not even consider immersion cooling in the context of datacom equipment.
Steam Degreaser With Open Tub
These devices are widely used in the world for precision cleaning of various parts, ranging from screws and bearings to orthopedic implants, printed circuit boards and injection nozzles.
Installation of steam degreasing company Reibesam.
The following article will be devoted to this technology.
The degreaser is an open rectangular tank with two rows of cooling coils installed on top of the perimeter. Up to a certain height, the tank is divided into two parts, or baths filled with a volatile solvent. In the first bath, the solvent is heated from the bottom and boils. Couples rise to the height of the first row of cooling coils, creating a saturated steam zone below this level. The condensate flows downwards and flows through the water separator into the rinse bath. Thus, as a result of distillation, only the solvent free from contamination enters this bath.
Scheme of the device steam degreaser.
These systems are capable of cleaning thousands of parts per shift, consuming a small amount of solvent. At the same time, most of the time they are either fully open or covered with a horizontally moving lid when not in use. Minimization of losses of solvent as a result of ablation is carried out at the expense of secondary cooling coils operating at temperatures below 0 ° C.
Open bath submersible cooling concept
This concept is based on the premise that electronic equipment can be cooled by immersion in a half-open bath, which is in many ways similar to the steam degreaser discussed above. The term “half-open” means that the bath is closed when access to equipment is not required and, as an example of such a design, a chest freezer for food can be given. Just like him, the bath operates at atmospheric pressure and does not have special hermetic connectors for supplied and discharged electricity.
In this system, each server or node is connected to the backplane at the bottom of the tub (as opposed to the back of the server rack). The bath is partially filled with volatile dielectric fluid.
Model open submersible cooling system.
Electrical wiring below the level of the liquid is brought into the channel, and out of it at the top of the tank. The built-in vapor condenser is cooled with either tap water or water used for comfortable heating (yes, there is such a concept in the USA) .
In addition, vapors can passively flow into the so-called natural draft cooling tower, giving off heat directly to the outside air, without using water as an intermediate coolant.
The Allied Control project with 3M fluoroketone is a two-phase immersion cooling in an open bath.
The concept has many advantages over traditional liquid cooling systems. The basic principle is that there is no need for most of the equipment that is mandatory for air and liquid cooling, and, accordingly, issues related to its installation, reliability and power consumption are not considered. The power density and reliability of the system is very high. Moreover, this technology is a priori inherent fire-resistant properties.
Of course, within this model, other aspects should be considered, for example, loss of working fluid as a result of ablation. But, since they occur in one particular place, and not in an innumerable number of connectors, they can easily be counted and reduced with the help of techniques that we will discuss later.
Inside the baths, heat from dozens of servers causes Novec fluid to boil.
Her vapors are condensed on a cooling coil and returned to the system.
Thermal efficiency
The thermal efficiency of the system consists of two components. The first is determined by the design of the printed circuit board and is quantitatively expressed in the temperature difference between the active layer in the processor chip, as the main element that needs to be cooled, and the temperature of the working fluid. The second component is determined by the temperature difference between the working fluid and the supplied water. In this case, the temperature of the working fluid, we mean its boiling point at atmospheric pressure.
The processor in its typical configuration in the form of a chip, a substrate and a heat-spreading lid with a built-in radiator is almost ideal for passive immersion cooling systems. In most cases, only a 100 micron layer of a porous metal coating is required, which improves heat transfer during boiling. These coatings provide a heat transfer coefficient of> 10 W / cm2 * K with a heat flux of 30 W / cm2.
If the coating operation is introduced already at the stage of the processor production, then there will be no need for a secondary thermal interface, commonly used in many liquid cooling schemes.
Model submersible 2-phase cooling system 3M.
The next point to consider is an assessment of how much power can be cooled by this system. In one study, the authors suggest that 100 cm3 of working fluid is required to cool a 1 kW module, provided that its configuration is known and taken into account, that is, the density of the components.
Conducted experiments
For the experiment, a model printed circuit board was assembled with 20 heat-generating ceramic elements with dimensions of 19 * 19 mm and a power of 200 W each. At the same time, on the one hand, a copper radiator of 30 * 30 * 3 mm is attached to them with epoxy glue, with the coating applied on the opposite side, increasing the heat transfer during boiling. Thermocouples in the working fluid and in each of the elements allow us to calculate the individual thermal resistance and make sure that the elements do not go into film boiling mode. Next, this model PCB is immersed in a narrow vertical container of the same shape with gaps of 4 and 7 mm between the boiling surface and the wall.
Boil Novec on a crypto farm.
During the experiment, it was shown that this configuration is capable of removing 4 kW of heat (200 W from each element) through a 4 mm gap at atmospheric pressure if Hydrofluoroether is used as the working fluid — C3F7OCH3. At the same time, 4 kW is equivalent to a heat flux of 11.7 W / cm2 against a flux of 1.7 W / cm2 observed in the Cray X1E supercomputer when cooled by spraying!
The results of the experiment suggest that a value of 1 kW of heat output per 100 cm3 of working fluid is definitely attainable. Also significantly reduced the amount of materials used and various kinds of emissions.
Chemistry of working liquids
The table below presents the properties of one hydrofluorine ester and two fluoroketones. They have the required thermal characteristics, are safe and compatible with various materials and have been tested in an open submersible cooling system.
Note that the dielectric characteristics of fluoroketones are similar to those of perfluorocarbon C6F14, which was often used in immersion cooling systems. At the same time, hydrofluoroether has a higher dielectric constant and lower resistance, which may limit its use in some cases. The first of the presented fluoroketones with a global warming potential value of only 1, is now widely used in the world as a fire extinguishing agent.
| Property | Working fluid | |||
| Molecular formula | C6F14 | C6F9OH5 | C6F12O | C7F14O |
| Type of | PFC (perfluorocarbon) | HPE (hydrofluoroether) | FC (fluoroketone) | FC (fluoroketone) |
| Boiling, ° C | 56 | 76 | 49 | 74 |
| Tzamerzaniya, ° C | <-100 | <-100 | <-100 | <-100 |
| Flash, ° | not | not | not | not |
| σ, mN / m | 12 | 13.6 | 10.8 | 12.3 |
| k, W / m * K | 0.057 | 0.068 | 0.059 | ~ 0.06 |
| Sid, J / kg * K | 1050 | 1220 | 1103 | 1130 |
| ρ, kg / m3 | 1680 | 1420 | 1600 | 1670 |
| ν, cSt | 0.4 | 0.41 | 0.4 | 0.52 |
| Rasysch. steam at 25 ° C, kPa | 30.9 | 15.7 | 40.4 | 15.7 |
| Rasysch. steam at 100 ° C, kPa | 350 | 206 | 441 | 228 |
| Specific resistance, GOM * m | 1,000,000 | 0.1 | 10,000 | 10,000 |
| Dielectric constant | 1.76 | 7.3 | 1.84 | 1.85 |
| Global warming potential | 9300 | 55 | one | one |
| MPC, ppm | Not determined | 200 | 150 | 150 |
System efficiency in terms of fluid loss
Indeed, this is the main factor affecting the viability of the presented system.
There are formulas for calculating fluid losses during filling, start-up and operation. They make it possible to conclude that the most effective measure to reduce losses as a result of the entrainment of vapors will be the creation of such a structure in which the ascending vapors will be captured using a secondary cooling coil, which automatically turns on when the system temperature exceeds the specified allowable temperature.
These racks contain mining ASIC boards immersed in Novec fluids (located in Hong Kong).
results
In the course of the experiments described above, it was shown that when using commercially available fluoroketone with a boiling point of 49 ° C in an open bath and a flow of water of 15 gallons (which is approximately equal to 3.785 * 15 = 56.8 liters) per minute, the temperature in the active layer of the chip processor will not exceed 60 ° C. It is enough to use water with a temperature of 28 ° C.
If it is acceptable to raise the temperature in the active layer of the processor to 83 ° C, and the volume flow of water will reach 30 gallons per minute, water can be used for primary coils with a temperature of 62 ° C.
The power density that the capacity is capable of cooling is 130 kW / m2, which is well above the 52 kW / m2 limit that is typical for racks with air or hybrid cooling. In the transition to a server scale with a completely liquid cooling system of this type, we get 25 kW / m2 against 2.2 kW / m2 for an air cooled server.
In addition, this system saves space, as it does not require the installation of additional equipment, mandatory for an air cooling system. Exhaust heat can be used for heating buildings, heating greenhouses and other objects.
Practical implementation
An example of the implementation of the new cooling concept can be found here .
Where else can you find out information?
On our site .
Where can I buy?
Call us at the multichannel phone +7 495 784 7474.
Ask questions in the comments, we will try to answer all!
Do you have anything else to read?
Of course!
" Dry Novec 1230 water to protect the server and not only
» How to safely read Habr at work using our information security screens
" From spray to film in one zilch. Paint Defender: protective film on the car body
» Tracing and e-marking technology for underground utilities. Part 1
If we made any mistakes, write to the LAN, we will fix everything promptly. Do not forget, we will correct the error, and your comment will remain hanging.
Thank you for your attention, we hope this article has been helpful to you!
legal information
3M and Novec are registered trademarks.
UPD: Thanks for pointing out the WarP Bitcoin farm.
Source: https://habr.com/ru/post/206658/
All Articles