Errors in the project of the data center that you feel only during the operation phase
Errors in the design of the data center is a time bomb. The data center will be built, they will be put into operation, and it will even work, but an hour will come, and problems will start to come out. In the best case, miscalculations in the project will cause inconvenience to the engineers of the operation service, at worst, they will question the work of the data center. Only now it will be too late to drink Borjomi: the builders with the signed acceptance certificate have already done a pen, and the project itself does not imply maneuvers.
We have selected the most bitter mistakes in the data center projects that we faced during eight years of designing, building and operating data centers. Open the hit parade.
Error number 1. Lowered design margin for energy consumption of engineering infrastructure equipment. Real consumption will be different from the passport. Is always. Yes, expensive vendors will forgive us, but most often in the documentation they lower the power consumption so that the equipment looks more economical.
')
The actual energy consumption can be calculated on the basis of the statistics collected for the already operating machine rooms. If there is no statistics, then throw a 10% passport data.
Error number 2. Insufficient number of backup machines from MSB to SCHR . A data center is a living organism that grows and changes. New equipment will need to be connected somewhere. If there is no space for backup automatics in switchboards, then additional boards will have to be installed, which is difficult when the data center is running, or to upgrade existing ones, which is expensive. Both options will require disconnecting the engine room from powering at least one beam.
At the design stage, if possible, lay a reserve for automata up to 30%. It is not necessary to immediately install additional machines, just leave space for them.
Error number 3. There is no possibility of phase balancing. Distribute the load evenly across the three phases of each beam. Control the currents in all phases on each element of the electrical network (MSB, switchboards, UPS) and, if necessary, transfer the load from one phase to another.
Error number 4. All fans of external units of direct expansion air conditioners are connected to the same phase. Often single-phase fans by inertia connected to the first phase. In order not to face the overload of the first phase, alternate: the first external unit to phase 1, the second to phase 2, the third to phase 3, the fourth to phase 1 again, and so on.
Error number 1. The type of cooling system does not take into account the architecture of the machine rooms. It is necessary to design refrigeration based on the architecture of the room. For example:
Error number 2. Improper air circulation. Because of this error, the cold in the machine room may be in excess, and the equipment will still overheat. If the cold air gets to the equipment through the raised floor, then its height should be sufficient: for a hall for 100 racks not less than 600 mm. Make sure that there are no obstacles in the form of pipes and improperly located power trays on the way of cold air under the raised floor.
In the machine room without a raised floor, you can use inter-rack air conditioning along with insulation of hot or cold corridors.
Similarly, we solve the problem in other rooms where cooling is required (room for the main switchboard, UPS).
Mistake number 3. Incorrectly calculated the actual temperature. Based on the recommended building codes for the region (SNiP), where it is planned to build a data center. This is correct, but we must make an amendment to the location of the real object. If the roof of the building is hot in the sun or there are problems with the free flow of air, the actual temperature will be several degrees higher. During operation, it stirs overheating of external units or chillers on hot summer days.
Error number 4. Not thought out to maintain the level of humidity. In machine rooms it is recommended to maintain humidity in the range of 30-70%. The type of cooling system affects the level of humidity: direct expansion air conditioners “dry” the air, so at the design stage lay the required number of steam humidifiers. When using the chiller system on “warm water”, on the contrary, we install air dryers.
Error number 5. There is no convenient access to chillers and external units. Convenience means round-the-clock and quick access. If you need to get a special permit for access to the roof with chillers, look for the person in charge with the keys, then this is a problem: any accident will require quick action.
This should not be a flimsy ladder that only a climber can climb. Chillers and external units need to be serviced and repaired, which means it will have to go up this ladder with tools. Separately, consider the method of delivery to the roof of large spare parts - fans, compressors.
On the roof, immediately provide outlets, fed through a protective shutdown device (RCD), and a water supply for washing chillers and outdoor units in the spring and in the season of poplar fluff.
Error number 6. Not taken into account the actual performance of freon air conditioners. Productivity of air conditioners depends on many factors, for example, on the length of freon lines. Actual performance is calculated in a special program provided by the vendor. This number and need to take when designing. Passport value is suitable only for the first estimation.
Here we stop and listen to a separate cry of a service engineer. Monitoring tasks are often not taken into account at the design stage: they are simply forgotten or deliberately refused for the sake of economy. After a couple of accidents, life still makes monitoring work. In the already working data center, building a monitoring system is more difficult and longer. And most importantly - all this time, the data center will work without control. The most terrible accidents are those that you don’t even guess. Here is our top 3:
Error number 1. Refusal of equipment monitoring modules. This is the very first thing on which they begin to save when they see the round value of the UPS or air conditioner. When, after all, decide to purchase them, you will have to beat the money again, wait for the delivery of the missing modules. All this time the life of the equipment will remain a mystery to you.
Separately, we note current analyzers in switchboards that collect information about the state of power supply systems in all sections of the route. At the design stage in advance, provide space for them in the switchboards.
Error number 2. Lack of documentation on the monitoring system. This documentation fixes:
These documents are drawn up in conjunction with the maintenance service at the design stage and are transmitted to it upon delivery of the data center. Without them, engineers will have to independently collect information about the principles of the monitoring system.
Error number 3. The absence or unsuccessful arrangement of temperature sensors and leakage sensors. For correct monitoring of the temperature in the engine room, we recommend to hang at least three sensors in each cold corridor (in the middle, beginning and end), one - in the hot. If the data center uses chiller cooling system, then provide leakage sensors under each element that can leak, and not just under the air conditioner.
Error number 1. The fire extinguishing system does not include warehouses and other additional premises. Due to a fire in the warehouse, you can lose a year's supply of spare parts and consumables. Even more dangerous is the prospect of moving fire from ancillary rooms to the computer room or rooms with engineering systems. When designing a fire extinguishing system, do not forget about warehouses and auxiliary premises.
Error number 2. Sirens / loudspeakers are not audible in engine rooms. The operation service when receiving the data center checks the warning system in empty halls, and the signal for evacuation is heard in almost complete silence. When the machine room is full, all the racks and air conditioners will work, then the volume of the sirens may not be enough. In the project, lay the power of the loudspeaker in such a way that in case of a fire it can “drown out” the filled hall.
Error number 3. The operation of the sensors according to the temperature in the container DGU. If using a container-type diesel generator set, make sure that the temperature fire detector does not overheat from the diesel generator set. Due to overheating, they will falsely trigger.
A similar problem occurs when an exhaust container from inside a nearby diesel generator set gets inside: the sensor falsely triggers on the exhaust. At the design stage, determine the optimal distance between the DGS in the group and the height of the exhaust pipe of each diesel generator.
Error number 1. Insufficient capacity of trunk cable trays and entry to the halls. Lay in the project reserve for cable entries in the machine room and the aisles between the floors. It will be difficult to drill new holes in walls or interfloor overlaps in an already operating machine room.
Error number 1. Too small doors and corridors. Yes, funny. But only as long as you do not have to disassemble the wall to make a broken air conditioner or UPS.
Tip from the cap: the door height must be such that the highest equipment passes through it, and the widest is the width. For the machine room it is racks and air conditioners.
Error number 2. Thresholds, differences in floor height in the data center. It is advisable to avoid large differences in floor height in the data center project. Heavy equipment (UPS, storage) is difficult to drag through sills. Some equipment does not tolerate inclinations at all: the vendor will refuse warranty service if the built-in equipment level records inclinations during transportation.
Error number 3. Difficult access and unloading for large vehicles. Organize convenient and round-the-clock access for unloading / loading of large equipment. In Moscow, for example, during the day the entry of trucks into the city is restricted. Equip a place of unloading with a lifting table for unloading equipment from machines of any height. Without it, every time you have to order a loader or crane.
Error number 4. Lack of freight elevators and lifts . For high-rise data center, provide elevators and lifts. Their dimensions are calculated on the basis of the largest equipment that we have to deliver to the upper floors.
Error number 5. Lack of support facilities. If warehouses are not provided in the data center, then very soon the whole spare parts kit, equipment boxes will be moved to the corridors, machine rooms and racks themselves. It is desirable that the warehouse was two:
Make sure that inside the data center there is a platform for unpacking equipment, a room where equipment brought from the cold can stand at room temperature.
Error number 6. There are no places to work with the equipment. Engineers will thank you for a separate area for working with equipment. In the engine rooms hot / cold and noisy. Equip this work area with a table, electrical and power sockets so that you can assemble the server (add a hard drive, fasten a sled, etc.) in more comfortable conditions.
Yes, it is also being developed at the design stage. If it is different, then most likely the following will turn out: one contractor designs the power supply, the other - the cold supply. Everyone uses only the well-known marking. Builders will have their own designations. When the data center is handed over, it will be difficult to deal with this legacy and understand where and what is located.
To prevent this from happening, a single label is created at the design stage, which is coordinated with the maintenance service. The labeling principle should be transparent: for example, a letter is an element type, the first digit is the number of the data center, the second is the hall number, the third is the element number. Then any service engineer will understand that under “K1.3.4” there are 4 air conditioners in the third hall in the data center 1.
The labeling of each element must be unique.
That's all. All the turns of fate in the project do not provide, but we hope that this list of errors will be useful to those who are going to design a server or data center.
Share in the comments with what project errors data centers or server faced you.
We have selected the most bitter mistakes in the data center projects that we faced during eight years of designing, building and operating data centers. Open the hit parade.
POWER SUPPLY
Error number 1. Lowered design margin for energy consumption of engineering infrastructure equipment. Real consumption will be different from the passport. Is always. Yes, expensive vendors will forgive us, but most often in the documentation they lower the power consumption so that the equipment looks more economical.
')
The actual energy consumption can be calculated on the basis of the statistics collected for the already operating machine rooms. If there is no statistics, then throw a 10% passport data.
Error number 2. Insufficient number of backup machines from MSB to SCHR . A data center is a living organism that grows and changes. New equipment will need to be connected somewhere. If there is no space for backup automatics in switchboards, then additional boards will have to be installed, which is difficult when the data center is running, or to upgrade existing ones, which is expensive. Both options will require disconnecting the engine room from powering at least one beam.
At the design stage, if possible, lay a reserve for automata up to 30%. It is not necessary to immediately install additional machines, just leave space for them.
Error number 3. There is no possibility of phase balancing. Distribute the load evenly across the three phases of each beam. Control the currents in all phases on each element of the electrical network (MSB, switchboards, UPS) and, if necessary, transfer the load from one phase to another.
Error number 4. All fans of external units of direct expansion air conditioners are connected to the same phase. Often single-phase fans by inertia connected to the first phase. In order not to face the overload of the first phase, alternate: the first external unit to phase 1, the second to phase 2, the third to phase 3, the fourth to phase 1 again, and so on.
COLD SUPPLY
Error number 1. The type of cooling system does not take into account the architecture of the machine rooms. It is necessary to design refrigeration based on the architecture of the room. For example:
- in machine rooms with a low ceiling (less than 3.5 m) and wide corridors (more than 20 racks fit between two air conditioners), it is better to use inter-rack (in-row), rather than cabinet air conditioners;
- For a high-rise building, a direct expansion (freon) system is most likely not suitable. The maximum length of freon tracks (the distance between the internal and external blocks) is not more than 50 meters horizontally, and if there is a rise, then even less;
- Chiller system will require a lot of space for pipelines, separate rooms for pumping stations, tanks, batteries.
Error number 2. Improper air circulation. Because of this error, the cold in the machine room may be in excess, and the equipment will still overheat. If the cold air gets to the equipment through the raised floor, then its height should be sufficient: for a hall for 100 racks not less than 600 mm. Make sure that there are no obstacles in the form of pipes and improperly located power trays on the way of cold air under the raised floor.
In the machine room without a raised floor, you can use inter-rack air conditioning along with insulation of hot or cold corridors.
Similarly, we solve the problem in other rooms where cooling is required (room for the main switchboard, UPS).
Mistake number 3. Incorrectly calculated the actual temperature. Based on the recommended building codes for the region (SNiP), where it is planned to build a data center. This is correct, but we must make an amendment to the location of the real object. If the roof of the building is hot in the sun or there are problems with the free flow of air, the actual temperature will be several degrees higher. During operation, it stirs overheating of external units or chillers on hot summer days.
Error number 4. Not thought out to maintain the level of humidity. In machine rooms it is recommended to maintain humidity in the range of 30-70%. The type of cooling system affects the level of humidity: direct expansion air conditioners “dry” the air, so at the design stage lay the required number of steam humidifiers. When using the chiller system on “warm water”, on the contrary, we install air dryers.
Error number 5. There is no convenient access to chillers and external units. Convenience means round-the-clock and quick access. If you need to get a special permit for access to the roof with chillers, look for the person in charge with the keys, then this is a problem: any accident will require quick action.
This should not be a flimsy ladder that only a climber can climb. Chillers and external units need to be serviced and repaired, which means it will have to go up this ladder with tools. Separately, consider the method of delivery to the roof of large spare parts - fans, compressors.
On the roof, immediately provide outlets, fed through a protective shutdown device (RCD), and a water supply for washing chillers and outdoor units in the spring and in the season of poplar fluff.
Error number 6. Not taken into account the actual performance of freon air conditioners. Productivity of air conditioners depends on many factors, for example, on the length of freon lines. Actual performance is calculated in a special program provided by the vendor. This number and need to take when designing. Passport value is suitable only for the first estimation.
MONITORING
Here we stop and listen to a separate cry of a service engineer. Monitoring tasks are often not taken into account at the design stage: they are simply forgotten or deliberately refused for the sake of economy. After a couple of accidents, life still makes monitoring work. In the already working data center, building a monitoring system is more difficult and longer. And most importantly - all this time, the data center will work without control. The most terrible accidents are those that you don’t even guess. Here is our top 3:
Error number 1. Refusal of equipment monitoring modules. This is the very first thing on which they begin to save when they see the round value of the UPS or air conditioner. When, after all, decide to purchase them, you will have to beat the money again, wait for the delivery of the missing modules. All this time the life of the equipment will remain a mystery to you.
Separately, we note current analyzers in switchboards that collect information about the state of power supply systems in all sections of the route. At the design stage in advance, provide space for them in the switchboards.
Error number 2. Lack of documentation on the monitoring system. This documentation fixes:
- list of installed sensors and equipment;
- the place of their installation;
- tracked parameters;
- wiring diagrams.
These documents are drawn up in conjunction with the maintenance service at the design stage and are transmitted to it upon delivery of the data center. Without them, engineers will have to independently collect information about the principles of the monitoring system.
Error number 3. The absence or unsuccessful arrangement of temperature sensors and leakage sensors. For correct monitoring of the temperature in the engine room, we recommend to hang at least three sensors in each cold corridor (in the middle, beginning and end), one - in the hot. If the data center uses chiller cooling system, then provide leakage sensors under each element that can leak, and not just under the air conditioner.
FIREFIGHTING
Error number 1. The fire extinguishing system does not include warehouses and other additional premises. Due to a fire in the warehouse, you can lose a year's supply of spare parts and consumables. Even more dangerous is the prospect of moving fire from ancillary rooms to the computer room or rooms with engineering systems. When designing a fire extinguishing system, do not forget about warehouses and auxiliary premises.
Error number 2. Sirens / loudspeakers are not audible in engine rooms. The operation service when receiving the data center checks the warning system in empty halls, and the signal for evacuation is heard in almost complete silence. When the machine room is full, all the racks and air conditioners will work, then the volume of the sirens may not be enough. In the project, lay the power of the loudspeaker in such a way that in case of a fire it can “drown out” the filled hall.
Error number 3. The operation of the sensors according to the temperature in the container DGU. If using a container-type diesel generator set, make sure that the temperature fire detector does not overheat from the diesel generator set. Due to overheating, they will falsely trigger.
A similar problem occurs when an exhaust container from inside a nearby diesel generator set gets inside: the sensor falsely triggers on the exhaust. At the design stage, determine the optimal distance between the DGS in the group and the height of the exhaust pipe of each diesel generator.
SKS AND TELECOM
Error number 1. Insufficient capacity of trunk cable trays and entry to the halls. Lay in the project reserve for cable entries in the machine room and the aisles between the floors. It will be difficult to drill new holes in walls or interfloor overlaps in an already operating machine room.
AUXILIARY PREMISES AND TERRITORY
Error number 1. Too small doors and corridors. Yes, funny. But only as long as you do not have to disassemble the wall to make a broken air conditioner or UPS.
Tip from the cap: the door height must be such that the highest equipment passes through it, and the widest is the width. For the machine room it is racks and air conditioners.
Error number 2. Thresholds, differences in floor height in the data center. It is advisable to avoid large differences in floor height in the data center project. Heavy equipment (UPS, storage) is difficult to drag through sills. Some equipment does not tolerate inclinations at all: the vendor will refuse warranty service if the built-in equipment level records inclinations during transportation.
Error number 3. Difficult access and unloading for large vehicles. Organize convenient and round-the-clock access for unloading / loading of large equipment. In Moscow, for example, during the day the entry of trucks into the city is restricted. Equip a place of unloading with a lifting table for unloading equipment from machines of any height. Without it, every time you have to order a loader or crane.
Error number 4. Lack of freight elevators and lifts . For high-rise data center, provide elevators and lifts. Their dimensions are calculated on the basis of the largest equipment that we have to deliver to the upper floors.
Error number 5. Lack of support facilities. If warehouses are not provided in the data center, then very soon the whole spare parts kit, equipment boxes will be moved to the corridors, machine rooms and racks themselves. It is desirable that the warehouse was two:
- operational with consumables and tools;
- long-term storage for large spare parts and empty boxes from under the equipment for warranty :)
Make sure that inside the data center there is a platform for unpacking equipment, a room where equipment brought from the cold can stand at room temperature.
Error number 6. There are no places to work with the equipment. Engineers will thank you for a separate area for working with equipment. In the engine rooms hot / cold and noisy. Equip this work area with a table, electrical and power sockets so that you can assemble the server (add a hard drive, fasten a sled, etc.) in more comfortable conditions.
MARKING EQUIPMENT
Yes, it is also being developed at the design stage. If it is different, then most likely the following will turn out: one contractor designs the power supply, the other - the cold supply. Everyone uses only the well-known marking. Builders will have their own designations. When the data center is handed over, it will be difficult to deal with this legacy and understand where and what is located.
To prevent this from happening, a single label is created at the design stage, which is coordinated with the maintenance service. The labeling principle should be transparent: for example, a letter is an element type, the first digit is the number of the data center, the second is the hall number, the third is the element number. Then any service engineer will understand that under “K1.3.4” there are 4 air conditioners in the third hall in the data center 1.
The labeling of each element must be unique.
That's all. All the turns of fate in the project do not provide, but we hope that this list of errors will be useful to those who are going to design a server or data center.
Share in the comments with what project errors data centers or server faced you.
Source: https://habr.com/ru/post/311826/
All Articles