The engineering device data center level TIER III, standing on four trunk lines
We have already talked about how our TIER III data center was designed and cost. It's time to show what happened.
First, on the site of the data center there was an open field, then a huge 100-meter trench was dug. Then the data center looked like a concrete platform, and even later - a hexagonal building was built from metal structures, inside which there are six data center modules (highlighted in green in the diagram), a “command center” for monitoring the backbone network throughout the country and an office (more on construction can be read in the publication "Data Center of our dreams in Yaroslavl: photos of construction and launch" ).
')
Carefully, traffic and gikporn from the Yaroslavl data center: 91 photos of the first module just launched, in the main building some final works are still underway, but there will be no more capital construction.
Territory
• 7 ha;
• 30 thousand sq. M. Area of ​​buildings.
Modular approach
• 6 independent data center modules (now installed first).
fault tolerance
• Tier III certification by the Uptime Institute.
Uninterrupted power
• 2 independent city inputs of 10 MW;
• Diesel dynamic uninterruptible power supply: 2500 kVA;
• Reservation 2N;
• Fuel storage 30 000 l, designed for 12 hours of operation of 6 modules with full load.
Cold supply
• N + 2;
• Water cooling on chiller installations;
• Cooling with outside air (Natural Free Cooling);
• Adiabatic cooling system;
• Uninterrupted power supply of cooling system equipment;
• Annual average PUE not more than 1.3.
Firefighting
• Novec gas is harmless to humans;
• Fire early warning system (VESDA).
Security
• Security 24x7x365;
• Access control system, video surveillance.
Environmental friendliness
• Uninterrupted power supply system without chemical batteries;
• Cooling with outside air up to 90% of the time in a year;
• Eco-friendly interior decoration materials office.
In reality, the building is almost as transparent as in the diagram, because a lot of polarized glass:
The main building where the data center is located
For comparison - here is a small container data center for seven racks, which we installed in three months: threw a couple of plates, and forward. The mobile data center and the neighboring blue container with DES worked at this point during commissioning and generally held a number of basic services until we raised the first computer hall.
Container Data Center
Here is the “cosmic cruise” scheme:
Object schema
The Hexagon in the middle is the main building where the Enterprise halls and office and service areas are located. The container data center is downstairs, it's small, and soon it will be taken away. And the two blocks of the "engines" of the cruiser are independent power installations (Power modules). Let's go there first.
Here is the building with power installations outside:
Power module where DDIBP is located
Diesel fuel is stored nearby in a blue container. In winter, we pour winter diesel, in summer - summer. There is no need to pump out anywhere - we generate diesel fuel at monthly maintenance checks.
Fuel storage capacity 30 000 l
But let's go inside the power module. We are met here is such a miracle device.
Motor-generator system for diesel-dynamic UPS
This is a whole range of devices. How it all works, we have already told at the design stage, here are the details (and the animation of the manufacturer). If it is very short - the power from two independent substations comes to us from the city, which is reserved by a system of diesel generators. In the middle of the circuit between the sources and the payload is a kinetic drive.
Scheme of organization of power supply complex
The data center will consume up to 10 MW. For obvious reasons, it is much more economical to take food from the city. But if something happens, you need to have either a huge supply of UPS batteries while the diesel is warming up, or a smaller supply of batteries and a pool with ice water for cooling, or your constantly operating power center (generation). One of the best solutions is DDIBP: a kinetic drive, plus a fast-start diesel, which is in standby mode. In practice, everything is much more complicated, but the general principle of operation is exactly that.
DDIBP control panel
It should be noted that the food from the city comes to us is not the highest quality. When it deteriorates or disappears, due to the energy of the kinetic accumulator (a seven-ton flywheel spun up to a tremendous speed). The energy of the residual rotation of the flywheel is enough with a margin to start the diesel even on the second attempt. Of the additional advantages - the same system makes it very easy to cope with the majority of power surges - roughly speaking, DDIBP in the middle of the circuit easily smoothes an average of about 80% of peaks and dips without turning on the motor-generator. Another advantage of using DDIBP is the absence of a problem with switching to a backup power system, since the kinetic drive is permanently connected to the circuit.
The fuel cleaning system, besides the usual filtration, also includes an impurity separation system, which performs fuel cleaning immediately before it is fed to the fuel pump.
Filter separator fuel supply system
Cylinder covers
Solenoid valve fuel supply system
One of our important tasks now is to guarantee energy efficiency with low consumption.
The pipe leads to the second floor, where the systems for plugging and cleaning of exhaust gases are located. In particular, the 4.5-meter two-ton silencers
Last time we promised to show the size of the silencers. Promised - we carry out:
At the DDIBP plant in Germany
This is a fuel tank.
The diesel for the DGS is stored in large quantities in the blue house outside, and in this tank - the necessary reserve for an hour or two of work (depending on the load). The tanks are connected, the fuel can be brought endlessly, and it will first be poured into the main storage, then - here, then - in the DSU itself. Of course, the second power module is the same, and there is its own autonomous fuel storage.
Discharge Ventilation Fan
Pipelines for diesel engine cooling system. A coolant circulates in pipes, which passes through radiators installed on the second floor.
To reduce losses, the flywheel rotates in helium gas. While it is in this environment, the cost of maintaining rotation, even at a wild speed, is minimal, that is, it is enough to invest energy in it for promotion, and then it consumes very little. Efficiency> 99%. Then, when necessary, the flywheel transfers the stored energy to the system.
>
Reducer-distributor supply of helium in the kinetic drive
Emergency stop valve of a diesel engine with a “fire” signal or emergency shutdown with a red button. It is used only as a last resort, and actually blocks the oxygen to the motor.
A fire is extinguished with Novec 1230 gas - at the concentration used it is not dangerous to humans. Our fire extinguishing system is smart: in the event of a fire on a diesel engine, it gives a chance to handle the fire independently. This increases the vitality of the data center.
Starter batteries
Power cables from the generator to the load (data center)
Fire alarm siren. As soon as it turns on - you need to run out of the room.
Control cabinet ventilation system
Automatic switch distribution box
Before you protection from fool. The lock closes critical panels. To press something, you need to remove the plastic curtain, turning it 180 degrees. Then push the button hard enough. Such protection options are made to ensure that nothing is done accidentally without a full awareness of what is happening.
The smoke removal system button is located in front of the entrance to the room
Here, in this unit, a test load is connected from the outside for routinely checking DDBP:
The load is connected by a special very flexible cable for its diameter
And here is the load itself. Tractor-rheostat. In the trailer there is something like a huge heater that effectively heats the air in order to take more power from the power module. It was not that the winter in Yaroslavl became warmer, but that they could have done it.
Tractor Rheostat
And these are two interesting objects. Lightning trap and lightning trap. More precisely, on the power module, a high mast structure is a passive lightning rod - a metal pin diverting the discharge deep underground. And then, in the center, on the roof of the data center, you can already see the active installation of a similar purpose, it allows you to achieve a similar effect only with significantly smaller sizes.
Passive and active ground outlets
The second module is symmetrical to the first. The only thing that distinguishes its surroundings is this additional object. This is a separate DES for the office, the requirements for uninterrupted power are much lower, which means that you do not need to spend the power DDBP.
DES for office
Now we go down in the metro to the Kremlin.
Cable collector
In fact, of course, this is a cable collector, but if you turn off the lights correctly, you can devote newcomers to the secrets of the USSR.
You can also do this:
Here are the cables as thick as the arm of a healthy engineer. Well, or with the leg of a pretty girl. For scraps of this cable was a real hunt for some installers at the construction site. According to their ideas, he is probably golden. They got a little, but they spoiled our nerves pretty much.
The cable, by the way, is mostly domestic, produced by the Kolchuginsky plant from the Vladimir region. There were no problems at our facility, but, of course, the cable should always be checked upon acceptance - it can be brought with a break, and with broken insulation, and a lot more.
"Glamor" fire fighting foam. When mounted, it expands and very tightly covers everything that can, to prevent the spread of fire and smoke through the holes, where, in this case, the cable passes.
So, there is food. The data center lacks more cooling and the Internet. And beer, but the brewery is right outside our window. We are fine with the transport network, there are two main communication rooms that reserve each other. 4 inputs of the main optics are connected to the data center, we have provided the possibility of installing 100Gbit / s transponders.
One day we will tell you about the device of our communication center in more detail.
Now we go to look at the cooling.
You can read more about cooling design here .
Again, briefly the main things:
Natural Free Cooling system organization chart
Freekuling works like this. In the "face" (mixing chamber) our data center is blowing in the wind. The cold air enters the main duct, gives the cold to the internal circuit, then flies farther through the data center and goes to the steppe. The hot environment of the machine room gives off heat through the heat exchanger to the outer space of the hangar in which the data center modules are located. Hot springs go up through the holes in the roof or forcibly through the mixing chamber. While simple, right ?
Layout of the main duct
The first one to meet us is the spider. This is a chiller. Spider we call him because of the characteristic shape.
Chiller
Why a chiller in a data center with free cooling? This is because if the external temperature rises above 24 degrees Celsius, the machinery will not be cooled with the desired efficiency. Let me remind you that we decided to build a data center in Yaroslavl also because it is cooler in summer (remember the night) than in Moscow, and you can save on cooling.
However, up to about a month a year, the average temperature rises above the indicated limit. In this case, we use the second cooling system on chillers that cool water in a huge network of pipelines. Again, simplifying to impropriety - these are large refrigerators that cool water. They require much more power, but nowhere to go. Reserving N + 2 chillers, this means three units are installed at the first machine hall.
Do you recognize these heroes?
Expansion tanks
There are 180 tons of water in the circuit.
Trunk manifold
Drain cock
Another spider, but the other side (on the side of the control panel)
Balancing valve
If emergency draining of water is required, it will flow into special pits.
Pumps provide circulation in the circuit. The second pump is a reserve.
Freonoprovoda lead to the roof. Bending is an oil-lifting hinge to ensure the return of oil to the compressor.
Soon we will finish with the glands and go to the gym. He is right behind this wall. But! First there is an important point. The fact is that the ambient air temperature is enough to cool the machinery with the necessary supply of only 8 months a year. And we run chillers for two to four weeks. Where are at least another three and a half months? Here, look at the device.
MNFC - heat exchanger
This is a heat exchange chamber, from here begins the cold corridor of the turbine hall, and here ends the hot. That is, in this device we make cold air out of hot air. This is a key element of the cooling system. The heat exchanger works in both the outside air cooling mode and the water cooling mode.
The main mode - a large amount of outdoor air is passed through the heat exchanger.
If the air is hot, we close the air intake and begin to cool the exchange chamber with a lead from the chiller, wasting energy on cooling.
Heat is exchanged behind this grate.
But when the air is warmer by only 3-4 degrees (up to 24 degrees), which happens in the "fallen out" months, we use life hacking from a school physics course. We simply saturate the air with moisture (we run it through the wall of water fog behind such a device), which gives the desired delta in the adiabatic process. This technology is much more economical than the chiller system, and is widely used in conjunction with free cooling.
Here, through, right through the building, passes the exhaust main duct. This is such a quite wide hot corridor.
Rooflights on the roof. The flaps open if necessary, and release hot streams of air.
All-all, we look mashzal. Now the rooms from the second to the sixth look like this: the modules have not yet been mounted.
But the first module has been launched and is now being filled. We arrived at the very moment when some of the servers are already running, but the hall is still half empty.
Data center module
Since the racks are typical, and the iron is generally homogeneous (this is our data center, and we know exactly what will be put here), it was possible to mount all the inputs and outputs at once.
Power sockets for connecting racks
Another plus of typical racks is the known dimensions of the cold and hot corridor. Here is the cold air blowing from the bottom of the raised floor. These panels are now covered with polyethylene - the corridor is not used, and we don’t want engineers to blow away anything. By the way, these cold corridors are perhaps the last reminder of why the admin is always in a sweater. Because it is very cold and blowing.
Cold air corridor
And this is our teleport device, a prototype.
But seriously - these are the doors that close (limit) the cold corridor so that the cooling works as efficiently as possible. This is how it looks in the part of the hall where the stands are already there.
Racks
Actually, the rack. The following equipment is now installed:
- Oracle servers of Hi-End and Mid-Range class;
- Storage system built on HDS, HP, IBM, Brocade hardware;
- The network core is built on Avaya hardware;
- HP servers.
Power distribution system for racks: busbars with power take-offs
We have a top-of-rack Panduit SKS system
Top-of-rack cassettes
Four cross-minus, soon to be five
Accuracy of installation - our small fad. Even in the Moscow test zone , where everything changes every day and the devil breaks the leg in the equipment, the installation is more or less accurate. We know from experience that an unsigned and carelessly mounted cable can cause a service interruption.
Switching Aggregation Switches
Active network equipment
10% of the hall are designed for high-density racks - up to 20 kW per unit. The most powerful racks are in the center of the room, there is a supply of cold because of the peculiarities of the movement of high-velocity air flow.
And here are the storage systems. They have not yet connected. Somewhere here will be stored data about your balance.
Tape library
These are fan heaters. With their help, we checked the room for the ability to remove heat. Yes, yes, just heated it and watched what happened.
Branch to the busline
Novec again, the stock lasts two fire extinguishing systems.
This is how the power cables that we saw in the basement go to the machine hall.
Now let's look at the office to admins and not only.
Main entrance to the data center building
We go around the data center and see temporary modular units - this is the dining room and the "office" - the places where the shift worked during the construction of the facility, when the office premises were not yet ready. The office is much better than in such a “container”.
Temporary office and dining room
At the entrance we have a data center layout:
Highlighted cooling circuit:
On the roof you can see the chiller condenser units:
You can also highlight the power and trunk inputs:
Before I get inside, I'll show the helmets in the dressing room. One of the builders turned out to be an artist and painted his helmet - and rushed.
Open space
The wall is made of school blackboard
But the rest room for change of administrators.
Coffee point
Cabinet
Bathroom
Back to the street again! All went around, but were not on the roof. And on the roof there are chillers, so they carefully peep:
Here is the roof itself. It is very large and striped. From the satellite, our data center can be seen as a circle logo.
We are approaching the outer chiller units.As from domestic air conditioners, just a little more.
These windows or skylights are the places where the hot air naturally leaves the building completely. Antiaircraft
lights
Each antiaircraft lamp has its own meteorological station, which sends data to the automation system.
And here is the final point of freonlines that we saw before.
Such a number of pipes, because each unit is dual-circuit, and there are two chillers for one (i.e., four-circuit chiller itself)
Elements building ventilation systems for general buildings
Here is the mast of lightning protection:
Here is our data center in Yaroslavl. While the first module of the flagship data center, designed for 236 racks, has been launched, we are slowly converting some of the “combat” systems there. By the way, the Federal Monitoring Center moved to us, and the girls from the unified service center will soon arrive. So we do not lose courage.
First, on the site of the data center there was an open field, then a huge 100-meter trench was dug. Then the data center looked like a concrete platform, and even later - a hexagonal building was built from metal structures, inside which there are six data center modules (highlighted in green in the diagram), a “command center” for monitoring the backbone network throughout the country and an office (more on construction can be read in the publication "Data Center of our dreams in Yaroslavl: photos of construction and launch" ).
')
Carefully, traffic and gikporn from the Yaroslavl data center: 91 photos of the first module just launched, in the main building some final works are still underway, but there will be no more capital construction.
Basic data center settings
Territory
• 7 ha;
• 30 thousand sq. M. Area of ​​buildings.
Modular approach
• 6 independent data center modules (now installed first).
fault tolerance
• Tier III certification by the Uptime Institute.
Uninterrupted power
• 2 independent city inputs of 10 MW;
• Diesel dynamic uninterruptible power supply: 2500 kVA;
• Reservation 2N;
• Fuel storage 30 000 l, designed for 12 hours of operation of 6 modules with full load.
Cold supply
• N + 2;
• Water cooling on chiller installations;
• Cooling with outside air (Natural Free Cooling);
• Adiabatic cooling system;
• Uninterrupted power supply of cooling system equipment;
• Annual average PUE not more than 1.3.
Firefighting
• Novec gas is harmless to humans;
• Fire early warning system (VESDA).
Security
• Security 24x7x365;
• Access control system, video surveillance.
Environmental friendliness
• Uninterrupted power supply system without chemical batteries;
• Cooling with outside air up to 90% of the time in a year;
• Eco-friendly interior decoration materials office.
In reality, the building is almost as transparent as in the diagram, because a lot of polarized glass:
The main building where the data center is located
For comparison - here is a small container data center for seven racks, which we installed in three months: threw a couple of plates, and forward. The mobile data center and the neighboring blue container with DES worked at this point during commissioning and generally held a number of basic services until we raised the first computer hall.
Container Data Center
Here is the “cosmic cruise” scheme:
Object schema
The Hexagon in the middle is the main building where the Enterprise halls and office and service areas are located. The container data center is downstairs, it's small, and soon it will be taken away. And the two blocks of the "engines" of the cruiser are independent power installations (Power modules). Let's go there first.
Here is the building with power installations outside:
Power module where DDIBP is located
Diesel fuel is stored nearby in a blue container. In winter, we pour winter diesel, in summer - summer. There is no need to pump out anywhere - we generate diesel fuel at monthly maintenance checks.
Fuel storage capacity 30 000 l
But let's go inside the power module. We are met here is such a miracle device.
Motor-generator system for diesel-dynamic UPS
This is a whole range of devices. How it all works, we have already told at the design stage, here are the details (and the animation of the manufacturer). If it is very short - the power from two independent substations comes to us from the city, which is reserved by a system of diesel generators. In the middle of the circuit between the sources and the payload is a kinetic drive.
Scheme of organization of power supply complex
The data center will consume up to 10 MW. For obvious reasons, it is much more economical to take food from the city. But if something happens, you need to have either a huge supply of UPS batteries while the diesel is warming up, or a smaller supply of batteries and a pool with ice water for cooling, or your constantly operating power center (generation). One of the best solutions is DDIBP: a kinetic drive, plus a fast-start diesel, which is in standby mode. In practice, everything is much more complicated, but the general principle of operation is exactly that.
DDIBP control panel
It should be noted that the food from the city comes to us is not the highest quality. When it deteriorates or disappears, due to the energy of the kinetic accumulator (a seven-ton flywheel spun up to a tremendous speed). The energy of the residual rotation of the flywheel is enough with a margin to start the diesel even on the second attempt. Of the additional advantages - the same system makes it very easy to cope with the majority of power surges - roughly speaking, DDIBP in the middle of the circuit easily smoothes an average of about 80% of peaks and dips without turning on the motor-generator. Another advantage of using DDIBP is the absence of a problem with switching to a backup power system, since the kinetic drive is permanently connected to the circuit.
The fuel cleaning system, besides the usual filtration, also includes an impurity separation system, which performs fuel cleaning immediately before it is fed to the fuel pump.
Filter separator fuel supply system
Cylinder covers
Solenoid valve fuel supply system
One of our important tasks now is to guarantee energy efficiency with low consumption.
The pipe leads to the second floor, where the systems for plugging and cleaning of exhaust gases are located. In particular, the 4.5-meter two-ton silencers
Last time we promised to show the size of the silencers. Promised - we carry out:
At the DDIBP plant in Germany
This is a fuel tank.
The diesel for the DGS is stored in large quantities in the blue house outside, and in this tank - the necessary reserve for an hour or two of work (depending on the load). The tanks are connected, the fuel can be brought endlessly, and it will first be poured into the main storage, then - here, then - in the DSU itself. Of course, the second power module is the same, and there is its own autonomous fuel storage.
Discharge Ventilation Fan
Pipelines for diesel engine cooling system. A coolant circulates in pipes, which passes through radiators installed on the second floor.
To reduce losses, the flywheel rotates in helium gas. While it is in this environment, the cost of maintaining rotation, even at a wild speed, is minimal, that is, it is enough to invest energy in it for promotion, and then it consumes very little. Efficiency> 99%. Then, when necessary, the flywheel transfers the stored energy to the system.
>Reducer-distributor supply of helium in the kinetic drive
Emergency stop valve of a diesel engine with a “fire” signal or emergency shutdown with a red button. It is used only as a last resort, and actually blocks the oxygen to the motor.
A fire is extinguished with Novec 1230 gas - at the concentration used it is not dangerous to humans. Our fire extinguishing system is smart: in the event of a fire on a diesel engine, it gives a chance to handle the fire independently. This increases the vitality of the data center.
Starter batteries
Power cables from the generator to the load (data center)
Fire alarm siren. As soon as it turns on - you need to run out of the room.
Control cabinet ventilation system
Automatic switch distribution box
Before you protection from fool. The lock closes critical panels. To press something, you need to remove the plastic curtain, turning it 180 degrees. Then push the button hard enough. Such protection options are made to ensure that nothing is done accidentally without a full awareness of what is happening.
The smoke removal system button is located in front of the entrance to the room
Here, in this unit, a test load is connected from the outside for routinely checking DDBP:
The load is connected by a special very flexible cable for its diameter
And here is the load itself. Tractor-rheostat. In the trailer there is something like a huge heater that effectively heats the air in order to take more power from the power module. It was not that the winter in Yaroslavl became warmer, but that they could have done it.
Tractor Rheostat
And these are two interesting objects. Lightning trap and lightning trap. More precisely, on the power module, a high mast structure is a passive lightning rod - a metal pin diverting the discharge deep underground. And then, in the center, on the roof of the data center, you can already see the active installation of a similar purpose, it allows you to achieve a similar effect only with significantly smaller sizes.
Passive and active ground outlets
The second module is symmetrical to the first. The only thing that distinguishes its surroundings is this additional object. This is a separate DES for the office, the requirements for uninterrupted power are much lower, which means that you do not need to spend the power DDBP.
DES for office
Now we go down in the metro to the Kremlin.
Cable collector
In fact, of course, this is a cable collector, but if you turn off the lights correctly, you can devote newcomers to the secrets of the USSR.
You can also do this:
Here are the cables as thick as the arm of a healthy engineer. Well, or with the leg of a pretty girl. For scraps of this cable was a real hunt for some installers at the construction site. According to their ideas, he is probably golden. They got a little, but they spoiled our nerves pretty much.
The cable, by the way, is mostly domestic, produced by the Kolchuginsky plant from the Vladimir region. There were no problems at our facility, but, of course, the cable should always be checked upon acceptance - it can be brought with a break, and with broken insulation, and a lot more.
"Glamor" fire fighting foam. When mounted, it expands and very tightly covers everything that can, to prevent the spread of fire and smoke through the holes, where, in this case, the cable passes.
So, there is food. The data center lacks more cooling and the Internet. And beer, but the brewery is right outside our window. We are fine with the transport network, there are two main communication rooms that reserve each other. 4 inputs of the main optics are connected to the data center, we have provided the possibility of installing 100Gbit / s transponders.
One day we will tell you about the device of our communication center in more detail.
Now we go to look at the cooling.
You can read more about cooling design here .
Again, briefly the main things:
Natural Free Cooling system organization chart
Freekuling works like this. In the "face" (mixing chamber) our data center is blowing in the wind. The cold air enters the main duct, gives the cold to the internal circuit, then flies farther through the data center and goes to the steppe. The hot environment of the machine room gives off heat through the heat exchanger to the outer space of the hangar in which the data center modules are located. Hot springs go up through the holes in the roof or forcibly through the mixing chamber. While simple, right ?
Layout of the main duct
The first one to meet us is the spider. This is a chiller. Spider we call him because of the characteristic shape.
Chiller
Why a chiller in a data center with free cooling? This is because if the external temperature rises above 24 degrees Celsius, the machinery will not be cooled with the desired efficiency. Let me remind you that we decided to build a data center in Yaroslavl also because it is cooler in summer (remember the night) than in Moscow, and you can save on cooling.
However, up to about a month a year, the average temperature rises above the indicated limit. In this case, we use the second cooling system on chillers that cool water in a huge network of pipelines. Again, simplifying to impropriety - these are large refrigerators that cool water. They require much more power, but nowhere to go. Reserving N + 2 chillers, this means three units are installed at the first machine hall.
Do you recognize these heroes?
Expansion tanks
There are 180 tons of water in the circuit.
Trunk manifold
Drain cock
Another spider, but the other side (on the side of the control panel)
Balancing valve
If emergency draining of water is required, it will flow into special pits.
Pumps provide circulation in the circuit. The second pump is a reserve.
Freonoprovoda lead to the roof. Bending is an oil-lifting hinge to ensure the return of oil to the compressor.
Soon we will finish with the glands and go to the gym. He is right behind this wall. But! First there is an important point. The fact is that the ambient air temperature is enough to cool the machinery with the necessary supply of only 8 months a year. And we run chillers for two to four weeks. Where are at least another three and a half months? Here, look at the device.
MNFC - heat exchanger
This is a heat exchange chamber, from here begins the cold corridor of the turbine hall, and here ends the hot. That is, in this device we make cold air out of hot air. This is a key element of the cooling system. The heat exchanger works in both the outside air cooling mode and the water cooling mode.
The main mode - a large amount of outdoor air is passed through the heat exchanger.
If the air is hot, we close the air intake and begin to cool the exchange chamber with a lead from the chiller, wasting energy on cooling.
Heat is exchanged behind this grate.
But when the air is warmer by only 3-4 degrees (up to 24 degrees), which happens in the "fallen out" months, we use life hacking from a school physics course. We simply saturate the air with moisture (we run it through the wall of water fog behind such a device), which gives the desired delta in the adiabatic process. This technology is much more economical than the chiller system, and is widely used in conjunction with free cooling.
Here, through, right through the building, passes the exhaust main duct. This is such a quite wide hot corridor.
Rooflights on the roof. The flaps open if necessary, and release hot streams of air.
All-all, we look mashzal. Now the rooms from the second to the sixth look like this: the modules have not yet been mounted.
But the first module has been launched and is now being filled. We arrived at the very moment when some of the servers are already running, but the hall is still half empty.
Data center module
Since the racks are typical, and the iron is generally homogeneous (this is our data center, and we know exactly what will be put here), it was possible to mount all the inputs and outputs at once.
Power sockets for connecting racks
Another plus of typical racks is the known dimensions of the cold and hot corridor. Here is the cold air blowing from the bottom of the raised floor. These panels are now covered with polyethylene - the corridor is not used, and we don’t want engineers to blow away anything. By the way, these cold corridors are perhaps the last reminder of why the admin is always in a sweater. Because it is very cold and blowing.
Cold air corridor
And this is our teleport device, a prototype.
But seriously - these are the doors that close (limit) the cold corridor so that the cooling works as efficiently as possible. This is how it looks in the part of the hall where the stands are already there.
Racks
Actually, the rack. The following equipment is now installed:
- Oracle servers of Hi-End and Mid-Range class;
- Storage system built on HDS, HP, IBM, Brocade hardware;
- The network core is built on Avaya hardware;
- HP servers.
Power distribution system for racks: busbars with power take-offs
We have a top-of-rack Panduit SKS system
Top-of-rack cassettes
Four cross-minus, soon to be five
Accuracy of installation - our small fad. Even in the Moscow test zone , where everything changes every day and the devil breaks the leg in the equipment, the installation is more or less accurate. We know from experience that an unsigned and carelessly mounted cable can cause a service interruption.
Switching Aggregation Switches
Active network equipment
10% of the hall are designed for high-density racks - up to 20 kW per unit. The most powerful racks are in the center of the room, there is a supply of cold because of the peculiarities of the movement of high-velocity air flow.
And here are the storage systems. They have not yet connected. Somewhere here will be stored data about your balance.
Tape library
These are fan heaters. With their help, we checked the room for the ability to remove heat. Yes, yes, just heated it and watched what happened.
Branch to the busline
Novec again, the stock lasts two fire extinguishing systems.
This is how the power cables that we saw in the basement go to the machine hall.
Now let's look at the office to admins and not only.
Main entrance to the data center building
We go around the data center and see temporary modular units - this is the dining room and the "office" - the places where the shift worked during the construction of the facility, when the office premises were not yet ready. The office is much better than in such a “container”.
Temporary office and dining room
At the entrance we have a data center layout:
Highlighted cooling circuit:
On the roof you can see the chiller condenser units:
You can also highlight the power and trunk inputs:
Before I get inside, I'll show the helmets in the dressing room. One of the builders turned out to be an artist and painted his helmet - and rushed.
Open space
The wall is made of school blackboard
But the rest room for change of administrators.
Coffee point
Cabinet
Bathroom
Back to the street again! All went around, but were not on the roof. And on the roof there are chillers, so they carefully peep:
Here is the roof itself. It is very large and striped. From the satellite, our data center can be seen as a circle logo.
We are approaching the outer chiller units.As from domestic air conditioners, just a little more.
These windows or skylights are the places where the hot air naturally leaves the building completely. Antiaircraft
lights
Each antiaircraft lamp has its own meteorological station, which sends data to the automation system.
And here is the final point of freonlines that we saw before.
Such a number of pipes, because each unit is dual-circuit, and there are two chillers for one (i.e., four-circuit chiller itself)
Elements building ventilation systems for general buildings
Here is the mast of lightning protection:
Here is our data center in Yaroslavl. While the first module of the flagship data center, designed for 236 racks, has been launched, we are slowly converting some of the “combat” systems there. By the way, the Federal Monitoring Center moved to us, and the girls from the unified service center will soon arrive. So we do not lose courage.
Source: https://habr.com/ru/post/253495/
All Articles