Excursion to the data center NORD-4
Today we will go for a walk on the largest commercial data center in Russia - NORD-4. Along the way we will tell you how everything is arranged in the TIER III data center.
The construction of the data center started in the summer of 2013 at the NORD site on Korovinskoe Highway in Moscow. NORD-4 was the largest project not only for DataLine, but also for the entire Russian market of commercial data centers. NORD-4 occupies a separate four-story building with a total area of ​​11,700 sq.m. 8 engine rooms accommodate 2016 stand. The total capacity of the data center is 18 MW.
')
The data center is notable not only for its scale, but for its embedded experience. We ourselves design, build and operate all our data centers since 2007, and NORD-4 has become the seventh for us.
NORD-4 was planned to be commissioned in stages, so the data center is designed so that the installation and commissioning of new lines do not interfere with the work of already running rooms.
In March 2015, the first phase of NORD-4 was put into operation - 2 halls with 252 staff, in December 2015 - the same number.
In June 2015, the NORD-4 project was certified by the Uptime Institute according to the TIER III Design standard, and in February 2016 we certified the already completed facility ( TIER III Facility ).
The next two queues of the data center at 1008 desk will start in 2017.
NORD-4 is located in the north of Moscow, on Korovinskoe Highway 41.
At the entrance to the data center we are greeted by a round-the-clock guard post. Like all of our data centers, NORD-4 is a secure object, so you can only get inside by means of an identity card and an application for a pass from a data center employee.
All the premises and the territory of the data center are viewed through the internal and external surveillance cameras. A picture of what is happening in NORD-4 is broadcast around the clock on the monitors of protection. All videos are recorded and stored for 3 months in case you need to play the events of the past days.
So, with a pass on the neck, boot covers on the legs and accompanied by an engineer on duty, we set off to explore the NORD-4. The data center itself can only be reached through a special compartment - the gateway. This kind of double barrage so that a person does not get into the data center without a permit. The gateway works like this: the visitor applies a pass to the access control reader (access control system), enters the gateway and closes the door behind him. After this, it repeats all the same actions with the second door leading to the data center. If in the gateway someone somehow turns out to be a person without a pass, the guard will block it and the second door will not open.
Each machine room is located in a separate container, with its own roof and walls. These internal buildings are called hermetic zones. They protect against leakage, dust ingress into machine rooms, provide the necessary degree of fire resistance.
Upon delivery of the data center, the maintenance service generously sheds the roof of the containment area with water and checks for leaks. For this test and real floods, the containment even has drain pipes.
Each machine room covers an area of ​​500 square meters. m and holds 252 rack. In 4 running halls 389 racks already work.
All racks with server and telecom equipment are installed on a raised floor 800 mm high. The main function of the raised floor is to supply cold air to the racks. Under the raised floor there are busbars, through which electricity is supplied to the racks, leakage and smoke detectors, sprayers of a fire extinguishing agent.
The second set of smoke detectors and fire extinguishing sprayers is located under the ceiling of the hall. Red Pipes is an early fire detection system (VESDA). She constantly "sniffs" the air and fixes the slightest smoke.
There are also speakers for warning of evacuation in case of fire.
In the event of a fire, Novec fire extinguishing agent is fed into the engine rooms through yellow pipes on the ceiling.
At the beginning and at the end of each row of racks, surveillance cameras are installed on the trays of the SCS. Separate 2 cameras record who enters and exits the hall.
If customers need an additional perimeter of physical protection, then we put their stands in special cells - cages. In the cage is set your access control system. Identification systems can be different: a magnetic card reader, fingerprints, retina. Inside you can hang additional CCTV cameras, volume sensors and movement.
The data center has 4 independent power centers. Each power center has 2 inputs from the city power grid, 2 step-down transformers, 4 diesel generators (DGU) and 11 uninterruptible power supplies (UPS). Each energy center feeds on 2 machine rooms and engineering equipment installed in them.
The distribution of electricity in the data center is approximately like this: the power from the city network is fed in two beams to the step-down transformers, and then to the main switchboard.
All critical engineering equipment has two power supply beams. All computing equipment is connected to two independent beams through uninterruptible power supplies (UPS).
If the city power is lost, the UPS automatically transfers its load to the batteries. At the same time, the automation system gives the command to start the DGS. They need about a minute to reach operating power, after which the data center begins to be powered by a diesel generator set, and not by batteries. Battery life at full load - 15 minutes.
NORD-4 employs 44 MGE Galaxy 7000 UPSs. The UPSs are reserved according to the 2N scheme.
This is how the rechargeable batteries look, at the expense of which the UPS feed the entire data center while the power plant is turned on.
At NORD-4, we use FG Wilson with a capacity of 1,750 kW. N + 1 redundancy: three DGUs are enough for one energy center to work, one is redundant.
On the territory of NORD-4 an underground fuel storage of 100 cu. m. From it, the fuel is automatically added to the consumable tanks DGU. On its own stocks without power from the city data center can work more than a day.
Tracks of the fuel storage, on which fuel enters the DGU, are reserved according to the 2N scheme .
Every two weeks we test the power supply system: for 1 hour we shut off one power supply beam and check the work of the diesel generator set, automation, UPS, monitoring systems and the training of engineers on duty.
In NORD-4, we settled on an ethylene glycol and water chiller scheme. In the machine rooms and power centers, there are internal units of air conditioners, chillers work on the roof, on the technical floor there are automation and control systems, heat exchangers, water circuit pumps, storage tanks.
In the circuit, which is located on the roof with chillers, ethylene glycol circulates. It does not freeze at sub-zero temperatures. Water flows through the cooling system pipes inside the data center.
Technical floor with cooling piping .
If ethylene glycol is warmer than the outside air, the cooling system removes unnecessary heat without turning on the compressors. Such a free cooling mode due to the “street” is called free cooling. NORD-4 switches to full free cooling when outside + 13 ° C and below, and into partial free cooling mode at a temperature of +14 to + 20 ° C.
With free-cooling, the power supply of the cooling system is much cheaper: when the air conditioning compressors are turned off, only fans and pumps consume electricity. Saves up to 85% of electricity.
According to the project, 14 chillers of the brand Stulz CyberCool-2 will be installed on the roof of the NORD-4. Now it works 7, since only half of the data center has been launched. Backup level N + 2.
Each NORD-4 machine room is cooled by 14 Stulz air conditioners. If you want to look inside the air conditioner, click here .
From the air conditioner, the cooled air enters the raised floor and through the perforated tiles of the raised floor enters the cold corridor. From there cold air is taken by the equipment. Warm air is blown out of the equipment into the hot aisle and returns to the air conditioner again. Such a scheme is called “cold and hot corridors”.
In the cold corridor, the temperature is maintained at 25 ° C (± 2), in the hot - by 10-15 degrees higher. Humidity - 30-70%.
This is how cold and hot air moves in the engine room arranged according to the principle of “cold and hot corridors”.
One of the cold corridors of the Whiskey Hall.
All components of the cold supply are reserved according to the scheme N + 1 or N + 2: any chiller, air conditioner, pump, heat exchanger, pipeline can be turned off for repair or service without interruption in the operation of the entire system.
On the 1st floor of the NORD-4 there is a fire extinguishing station with cylinders with Novec. This substance leaves the equipment and even paper dry when spraying. Because of this, Novec properties are also called “dry water”. Some craftsmen have adapted this substance for liquid cooling servers .
The system is automatic, but we are exploiting it in a semi-manual mode: automation only works when people are evacuated from the premises affected by fire. In the event of smoke, a sensor is triggered, which sends a signal to the security post indicating the place of fire. To prevent Novec from filing a false alarm, the guard checks the problem room. At the same time, in all the premises of the data center, a siren begins to howl, urging everyone to evacuate from the data center. At the time of the evacuation of ACS on all doors off, so that people can quickly leave the building.
If there is a fire, the guard checks to see if there is any people left inside, closes the door to the room and turns on the automation. After 45 seconds, Novec begins to flow into the room. After 2-3 minutes, already two guards in gas masks (one enters the hall, the other insures at the entrance) check whether it was possible to eliminate the fire.
Through these pipes, Novec rushes into machine rooms and engineering rooms. The fire extinguishing system is reserved according to the 2N scheme.
All information on the state of engineering infrastructure of data centers is collected in a single monitoring system. Each data center has an operational control center with a video wall, which displays detailed information on this data center in the form of visual diagrams, tables and graphs. The system was made on its own on the basis of Nagios and Cacti software. Be sure to tell about it in more detail in future posts.
In the operational control center of the data center there is a duty shift of engineers around the clock. At any one time, there is at least one engineer behind the video wall. The attendants monitor the main vital signs of the data center, work with client and infrastructure incidents.
In NORD-4, we monitor 1785 facilities and 15,200 engineering infrastructure parameters. Here are the main ones:
The state of the equipment (on / off, the presence of errors, loading, operating parameters): air conditioners, chillers, control panels, pumps, diesel power plants, UPS, batteries, leakage sensors, etc.
Climatic conditions : temperature and humidity in machine rooms and infrastructure facilities, leakage sensors, glycol temperature.
On the territory of the site NORD even its own weather station is operating. Collected statistics on weather conditions helps us to predict the load on the cooling system.
Power supply (voltage, current, frequency, power on current analyzers): inputs, main and intermediate switchboards, AVR and PDU in racks.
Physical security : ACS, outdoor and indoor security cameras.
On the video wall in the monitoring center displays the key parameters that determine the performance of the data center. All other data remain in the unified monitoring system and get on a special monitor of the video wall only in case of failures.
If one power supply is disconnected on this circuit diagram, we will see if the power supply to the UPS has passed, the battery charge level is on and the diesel generator set is on, the temperature in the hall has not increased.
Now only part of the data center is in operation, so the video wall in the MCC is still more modest than in our other data centers.
For comparison: this is how the MCC NORD 2 data center looks.
In addition to automated monitoring, twice a day, on-duty engineers bypass all machine rooms and infrastructure facilities with electronic checklists. If any client equipment is connected incorrectly , a light bulb burns out in the corridor, or the cross- connection is incorrectly set up somewhere, during the round it will be noted and entered as incidents in the ticket system. For the next shift and production director, the inspection results will come in the form of an automatically generated report.
Some of the parameters of the engineering systems are displayed on the website in the data center online section and in the DataLine4All mobile application for Android and iOS devices . More detailed information is available to customers of the data center in the Dashboard.
Finally, important things that add comfort when working in the NORD-4.
Unloading area All equipment enters the data center through a special compartment. Lifting table helps unload equipment boxes from trucks of any height.
For transportation of equipment in machine rooms on the 2nd and 3rd floors there are 2 freight elevators for 5 tons.
On the floors with machine rooms installed lockers, locked with a key for storing customer tools, parts and other trifles. This allows customers to conveniently store their treasures without using the rack.
We looked at all the basic infrastructure facilities of the NORD-4 data center. Ask questions in the comments if you want more details. You can see the working data center and get useful knowledge about the operation of the data center at one of the seminars of our free course “Data Center Engineering Infrastructure Management ”. Apply online , wait for our confirmation and come.
In the next posts we will describe in detail how the power supply in the data center is arranged.
A bit of history
The construction of the data center started in the summer of 2013 at the NORD site on Korovinskoe Highway in Moscow. NORD-4 was the largest project not only for DataLine, but also for the entire Russian market of commercial data centers. NORD-4 occupies a separate four-story building with a total area of ​​11,700 sq.m. 8 engine rooms accommodate 2016 stand. The total capacity of the data center is 18 MW.
')
The data center is notable not only for its scale, but for its embedded experience. We ourselves design, build and operate all our data centers since 2007, and NORD-4 has become the seventh for us.
NORD-4 was planned to be commissioned in stages, so the data center is designed so that the installation and commissioning of new lines do not interfere with the work of already running rooms.
In March 2015, the first phase of NORD-4 was put into operation - 2 halls with 252 staff, in December 2015 - the same number.
In June 2015, the NORD-4 project was certified by the Uptime Institute according to the TIER III Design standard, and in February 2016 we certified the already completed facility ( TIER III Facility ).
The next two queues of the data center at 1008 desk will start in 2017.
NORD-4 is located in the north of Moscow, on Korovinskoe Highway 41.
Physical security
At the entrance to the data center we are greeted by a round-the-clock guard post. Like all of our data centers, NORD-4 is a secure object, so you can only get inside by means of an identity card and an application for a pass from a data center employee.
All the premises and the territory of the data center are viewed through the internal and external surveillance cameras. A picture of what is happening in NORD-4 is broadcast around the clock on the monitors of protection. All videos are recorded and stored for 3 months in case you need to play the events of the past days.
So, with a pass on the neck, boot covers on the legs and accompanied by an engineer on duty, we set off to explore the NORD-4. The data center itself can only be reached through a special compartment - the gateway. This kind of double barrage so that a person does not get into the data center without a permit. The gateway works like this: the visitor applies a pass to the access control reader (access control system), enters the gateway and closes the door behind him. After this, it repeats all the same actions with the second door leading to the data center. If in the gateway someone somehow turns out to be a person without a pass, the guard will block it and the second door will not open.
Machine rooms
Each machine room is located in a separate container, with its own roof and walls. These internal buildings are called hermetic zones. They protect against leakage, dust ingress into machine rooms, provide the necessary degree of fire resistance.
Upon delivery of the data center, the maintenance service generously sheds the roof of the containment area with water and checks for leaks. For this test and real floods, the containment even has drain pipes.
Each machine room covers an area of ​​500 square meters. m and holds 252 rack. In 4 running halls 389 racks already work.
All racks with server and telecom equipment are installed on a raised floor 800 mm high. The main function of the raised floor is to supply cold air to the racks. Under the raised floor there are busbars, through which electricity is supplied to the racks, leakage and smoke detectors, sprayers of a fire extinguishing agent.
The second set of smoke detectors and fire extinguishing sprayers is located under the ceiling of the hall. Red Pipes is an early fire detection system (VESDA). She constantly "sniffs" the air and fixes the slightest smoke.
There are also speakers for warning of evacuation in case of fire.
In the event of a fire, Novec fire extinguishing agent is fed into the engine rooms through yellow pipes on the ceiling.
At the beginning and at the end of each row of racks, surveillance cameras are installed on the trays of the SCS. Separate 2 cameras record who enters and exits the hall.
If customers need an additional perimeter of physical protection, then we put their stands in special cells - cages. In the cage is set your access control system. Identification systems can be different: a magnetic card reader, fingerprints, retina. Inside you can hang additional CCTV cameras, volume sensors and movement.
Power supply
The data center has 4 independent power centers. Each power center has 2 inputs from the city power grid, 2 step-down transformers, 4 diesel generators (DGU) and 11 uninterruptible power supplies (UPS). Each energy center feeds on 2 machine rooms and engineering equipment installed in them.
The distribution of electricity in the data center is approximately like this: the power from the city network is fed in two beams to the step-down transformers, and then to the main switchboard.
All critical engineering equipment has two power supply beams. All computing equipment is connected to two independent beams through uninterruptible power supplies (UPS).
If the city power is lost, the UPS automatically transfers its load to the batteries. At the same time, the automation system gives the command to start the DGS. They need about a minute to reach operating power, after which the data center begins to be powered by a diesel generator set, and not by batteries. Battery life at full load - 15 minutes.
NORD-4 employs 44 MGE Galaxy 7000 UPSs. The UPSs are reserved according to the 2N scheme.
This is how the rechargeable batteries look, at the expense of which the UPS feed the entire data center while the power plant is turned on.
At NORD-4, we use FG Wilson with a capacity of 1,750 kW. N + 1 redundancy: three DGUs are enough for one energy center to work, one is redundant.
On the territory of NORD-4 an underground fuel storage of 100 cu. m. From it, the fuel is automatically added to the consumable tanks DGU. On its own stocks without power from the city data center can work more than a day.
Tracks of the fuel storage, on which fuel enters the DGU, are reserved according to the 2N scheme .
Every two weeks we test the power supply system: for 1 hour we shut off one power supply beam and check the work of the diesel generator set, automation, UPS, monitoring systems and the training of engineers on duty.
Cold supply
In NORD-4, we settled on an ethylene glycol and water chiller scheme. In the machine rooms and power centers, there are internal units of air conditioners, chillers work on the roof, on the technical floor there are automation and control systems, heat exchangers, water circuit pumps, storage tanks.
In the circuit, which is located on the roof with chillers, ethylene glycol circulates. It does not freeze at sub-zero temperatures. Water flows through the cooling system pipes inside the data center.
Technical floor with cooling piping .
If ethylene glycol is warmer than the outside air, the cooling system removes unnecessary heat without turning on the compressors. Such a free cooling mode due to the “street” is called free cooling. NORD-4 switches to full free cooling when outside + 13 ° C and below, and into partial free cooling mode at a temperature of +14 to + 20 ° C.
With free-cooling, the power supply of the cooling system is much cheaper: when the air conditioning compressors are turned off, only fans and pumps consume electricity. Saves up to 85% of electricity.
According to the project, 14 chillers of the brand Stulz CyberCool-2 will be installed on the roof of the NORD-4. Now it works 7, since only half of the data center has been launched. Backup level N + 2.
Each NORD-4 machine room is cooled by 14 Stulz air conditioners. If you want to look inside the air conditioner, click here .
From the air conditioner, the cooled air enters the raised floor and through the perforated tiles of the raised floor enters the cold corridor. From there cold air is taken by the equipment. Warm air is blown out of the equipment into the hot aisle and returns to the air conditioner again. Such a scheme is called “cold and hot corridors”.
In the cold corridor, the temperature is maintained at 25 ° C (± 2), in the hot - by 10-15 degrees higher. Humidity - 30-70%.
This is how cold and hot air moves in the engine room arranged according to the principle of “cold and hot corridors”.
One of the cold corridors of the Whiskey Hall.
All components of the cold supply are reserved according to the scheme N + 1 or N + 2: any chiller, air conditioner, pump, heat exchanger, pipeline can be turned off for repair or service without interruption in the operation of the entire system.
Fire safety
On the 1st floor of the NORD-4 there is a fire extinguishing station with cylinders with Novec. This substance leaves the equipment and even paper dry when spraying. Because of this, Novec properties are also called “dry water”. Some craftsmen have adapted this substance for liquid cooling servers .
The system is automatic, but we are exploiting it in a semi-manual mode: automation only works when people are evacuated from the premises affected by fire. In the event of smoke, a sensor is triggered, which sends a signal to the security post indicating the place of fire. To prevent Novec from filing a false alarm, the guard checks the problem room. At the same time, in all the premises of the data center, a siren begins to howl, urging everyone to evacuate from the data center. At the time of the evacuation of ACS on all doors off, so that people can quickly leave the building.
If there is a fire, the guard checks to see if there is any people left inside, closes the door to the room and turns on the automation. After 45 seconds, Novec begins to flow into the room. After 2-3 minutes, already two guards in gas masks (one enters the hall, the other insures at the entrance) check whether it was possible to eliminate the fire.
Through these pipes, Novec rushes into machine rooms and engineering rooms. The fire extinguishing system is reserved according to the 2N scheme.
Monitoring center
All information on the state of engineering infrastructure of data centers is collected in a single monitoring system. Each data center has an operational control center with a video wall, which displays detailed information on this data center in the form of visual diagrams, tables and graphs. The system was made on its own on the basis of Nagios and Cacti software. Be sure to tell about it in more detail in future posts.
In the operational control center of the data center there is a duty shift of engineers around the clock. At any one time, there is at least one engineer behind the video wall. The attendants monitor the main vital signs of the data center, work with client and infrastructure incidents.
In NORD-4, we monitor 1785 facilities and 15,200 engineering infrastructure parameters. Here are the main ones:
The state of the equipment (on / off, the presence of errors, loading, operating parameters): air conditioners, chillers, control panels, pumps, diesel power plants, UPS, batteries, leakage sensors, etc.
Climatic conditions : temperature and humidity in machine rooms and infrastructure facilities, leakage sensors, glycol temperature.
On the territory of the site NORD even its own weather station is operating. Collected statistics on weather conditions helps us to predict the load on the cooling system.
Power supply (voltage, current, frequency, power on current analyzers): inputs, main and intermediate switchboards, AVR and PDU in racks.
Physical security : ACS, outdoor and indoor security cameras.
On the video wall in the monitoring center displays the key parameters that determine the performance of the data center. All other data remain in the unified monitoring system and get on a special monitor of the video wall only in case of failures.
If one power supply is disconnected on this circuit diagram, we will see if the power supply to the UPS has passed, the battery charge level is on and the diesel generator set is on, the temperature in the hall has not increased.
Now only part of the data center is in operation, so the video wall in the MCC is still more modest than in our other data centers.
For comparison: this is how the MCC NORD 2 data center looks.
In addition to automated monitoring, twice a day, on-duty engineers bypass all machine rooms and infrastructure facilities with electronic checklists. If any client equipment is connected incorrectly , a light bulb burns out in the corridor, or the cross- connection is incorrectly set up somewhere, during the round it will be noted and entered as incidents in the ticket system. For the next shift and production director, the inspection results will come in the form of an automatically generated report.
Some of the parameters of the engineering systems are displayed on the website in the data center online section and in the DataLine4All mobile application for Android and iOS devices . More detailed information is available to customers of the data center in the Dashboard.
Convenience for operation
Finally, important things that add comfort when working in the NORD-4.
Unloading area All equipment enters the data center through a special compartment. Lifting table helps unload equipment boxes from trucks of any height.
For transportation of equipment in machine rooms on the 2nd and 3rd floors there are 2 freight elevators for 5 tons.
On the floors with machine rooms installed lockers, locked with a key for storing customer tools, parts and other trifles. This allows customers to conveniently store their treasures without using the rack.
We looked at all the basic infrastructure facilities of the NORD-4 data center. Ask questions in the comments if you want more details. You can see the working data center and get useful knowledge about the operation of the data center at one of the seminars of our free course “Data Center Engineering Infrastructure Management ”. Apply online , wait for our confirmation and come.
In the next posts we will describe in detail how the power supply in the data center is arranged.
Source: https://habr.com/ru/post/309520/
All Articles