VKontakte data center
Everyone knows that the heart of VKontakte is Zinger’s house on Nevsky Prospect. Today we will tell and show where its brain is and how it looks - the data center ITSVA.
A data center (data storage and processing center, also known as a data center) is a combination of several infrastructure systems that ensure the reliability and resiliency of server and network equipment.
')
You can not just put a bunch of servers and switches. It is necessary to create and maintain optimal conditions for their work. If you want to build your data center, you will need:
These are the basic life support systems, the very minimum that is needed just to run the equipment. But for this full-fledged data center you need something else. Namely:
We believe that you will have an excellent data center. In the meantime, let's see what ours looks like.
Why Icwa, what is the name? ITSVA is a research center of high-voltage apparatus, which used to be located in the building of the data center and worked for the benefit of the energy industry. We inherited a dystopian hangar with ceilings on the fifth floor and mysterious rooms with a meter-thick walls.
There are 640 racks in four machine halls - more than 20,000 servers and more than 200 switches, routers and DWDM systems with a capacity of more than 4 TB / s. The ASR9000 router with serial number 1 is installed here - at one time it was the first commercial installation of such a device in the whole world.
At its peak, the data center generates more than 1TB / c of external traffic. More than 10 of the largest international providers and international traffic exchange centers, as well as about 40 major operators of the Russian Federation are connected to our DWDM systems.
The first machine room. Perhaps this is where your favorite video lies.
All elements of the power supply system are reserved at least N + 1. Literally in front of the data center building is the Vostochnaya substation, from which power is supplied to the data center via two 6kV inputs. Further, through the distribution substation and automatic input of the reserve, the power is supplied through two independent inputs. Here is how it looks in the diagram (for simplicity, with one of four machine rooms):
Normal power supply scheme
Each node is duplicated and normally works under half load. In the event of an accident, the power will reach the engine room bypassing the failed section. For example, we lost one 6kV input:
Power supply in case of input failure
If everything is completely bad, and it is not necessary to count on inputs from the backbone network, uninterruptible power supplies come into play. Their task is to provide power to the engine rooms for a short time while the diesel generator sets are running.
Uninterruptible Power Supply
And these batteries are for them - outwardly very similar to automobile ones ...
... only occupy a few large rooms
Diesel generator sets (DGS) support life in the data center in the event of a prolonged accident or planned work in the power supply system. In addition to the fuel tanks in the DGU themselves installed an automatic container filling station of large capacity. Fuel from the tank is fed automatically to all DGI, the reserve is calculated for at least a day. If necessary, a diesel fuel truck will arrive within two hours.
Diesel generator sets
Each server and each switch is connected to two power inputs. As a rule, in modern equipment such an opportunity is provided by the manufacturer. For servers with one input, the power is duplicated using this device:
Static load switch + manual bypass Digital Energy
For a comfortable life of the equipment in the machine rooms you need to maintain a certain temperature range. That is why companies around the world are increasingly building their data centers somewhere near the Arctic Circle - in such conditions you can use the outside air to cool the servers. This is called “free cooling”, and this approach is considered to be the most energy efficient (why waste energy on cooling warm air, if you can take the cold air right away?).
We also use freecooling, but with some reservations. Despite the legendary Petersburg coolness, in summer the air temperature still sometimes rises above the cherished 20-25 °, and it has to be further cooled. In winter, by contrast, the air is too cold to be used immediately. In addition to the fact that servers can simply overcool, a change in temperature will lead to a shift in the dew point - and condensate is contraindicated in such equipment. In addition, the air with such a scheme gets from the street, which means that it must be further cleaned.
Free cooling is used in one of the four machine rooms; in the rest, the cooling system operates according to the classical scheme using precision air conditioners.
Indoor units of precision air conditioners
External units of precision air conditioners
Cold air from the mixing chamber or air conditioner is fed through a raised floor or air duct into the so-called “cold corridor”. This is an isolated space between the front sides of two rows of racks. Here it is:
Cold hall of the machine room
On the reverse side, the exhaust heated air enters the “hot corridor” - and from there it is sent back for refrigeration by freon to the indoor units of air conditioners. Thus, the circulation of clean (dust-free) air in the engine room is achieved.
Hot hall of the machine room
Kilometers carefully laid wires. No words are needed here.
In our data center gas fire extinguishing system is provided. Actually gas (freon) is stored under pressure in cylinders. In the event of a fire on the signal from the sensor in the engine room, the valve will be activated, and the gas through the pipes will flow to the source of ignition.
Cylinders with freon
Manometer
All health indicators of the data center are monitored in real time. These are temperature (from sensors of equipment and indoors), power supply, load on network equipment - the data is displayed on the attendant displays and controlled by automation. If something went wrong, the monitoring system itself will send a message about the problem to the engineers (VKontakte and SMS).
Power supply monitoring
Cooling system monitoring
Only employees can enter the territory; any door is equipped with an electronic lock with an access card reader. ITSVA is guarded on a 24/7 basis, video surveillance is conducted in each room.
ITSVA has a very good location: just a few kilometers from the hometown of VKontakte and near a reliable source of electricity.
Here, the process of upgrading equipment and increasing energy efficiency continues. PUE (Power Usage Effectiveness), or the coefficient of energy efficiency is a key indicator of data center assessment. It is considered as the ratio of all energy consumed by the data center, and the actual consumption of servers and network equipment. As is clear from this definition, the PUE of an ideal data center in a vacuum is 1.0. ITSVA is not an ideal data center in a vacuum, but we are systematically working on reducing this indicator.
The team of ICVA staff and network engineers on VKontakte is doing everything so that you can enjoy your favorite videos every day, watch new photos of friends and not worry about how complex infrastructure is used.
If you are a first-class specialist, you have a chance to get acquainted with our data center, becoming part of the team. We invite to work the head of the test lab of the data center and system administrators .
PS Articles on the technical side of VKontakte can also be read in our blog in Russian and English. You can ask the author in the official community .
How to build a data center?
A data center (data storage and processing center, also known as a data center) is a combination of several infrastructure systems that ensure the reliability and resiliency of server and network equipment.
')
You can not just put a bunch of servers and switches. It is necessary to create and maintain optimal conditions for their work. If you want to build your data center, you will need:
- Power supply system. Everything is clear here - servers run on electricity, and there are many of them. Therefore, the usual outlet for 220V, most likely, will not be enough.
- Cooling system. Even a gaming video card in an advanced computer requires a powerful cooler. What to say about the hundreds and thousands of high-performance devices.
- Structured Cabling System (SCS). Something must unite all the elements in a single unit. You will need a lot of cables and an ardent love for the process of pedanting them.
These are the basic life support systems, the very minimum that is needed just to run the equipment. But for this full-fledged data center you need something else. Namely:
- Fire extinguishing system. It is important to take care that a random spark does not turn your new data center into ruins.
- Access Control System (ACS). Do not leave the door open to all comers.
- Monitoring system. You need to know in time if something went wrong.
- Security alarm. In case if instead of a pass someone decides to use scrap.
- Video surveillance system.
We believe that you will have an excellent data center. In the meantime, let's see what ours looks like.
Welcome to ICVA
Why Icwa, what is the name? ITSVA is a research center of high-voltage apparatus, which used to be located in the building of the data center and worked for the benefit of the energy industry. We inherited a dystopian hangar with ceilings on the fifth floor and mysterious rooms with a meter-thick walls.
There are 640 racks in four machine halls - more than 20,000 servers and more than 200 switches, routers and DWDM systems with a capacity of more than 4 TB / s. The ASR9000 router with serial number 1 is installed here - at one time it was the first commercial installation of such a device in the whole world.
At its peak, the data center generates more than 1TB / c of external traffic. More than 10 of the largest international providers and international traffic exchange centers, as well as about 40 major operators of the Russian Federation are connected to our DWDM systems.
The first machine room. Perhaps this is where your favorite video lies.
Power supply
All elements of the power supply system are reserved at least N + 1. Literally in front of the data center building is the Vostochnaya substation, from which power is supplied to the data center via two 6kV inputs. Further, through the distribution substation and automatic input of the reserve, the power is supplied through two independent inputs. Here is how it looks in the diagram (for simplicity, with one of four machine rooms):
Normal power supply scheme
Each node is duplicated and normally works under half load. In the event of an accident, the power will reach the engine room bypassing the failed section. For example, we lost one 6kV input:
Power supply in case of input failure
If everything is completely bad, and it is not necessary to count on inputs from the backbone network, uninterruptible power supplies come into play. Their task is to provide power to the engine rooms for a short time while the diesel generator sets are running.
Uninterruptible Power Supply
And these batteries are for them - outwardly very similar to automobile ones ...
... only occupy a few large rooms
Diesel generator sets (DGS) support life in the data center in the event of a prolonged accident or planned work in the power supply system. In addition to the fuel tanks in the DGU themselves installed an automatic container filling station of large capacity. Fuel from the tank is fed automatically to all DGI, the reserve is calculated for at least a day. If necessary, a diesel fuel truck will arrive within two hours.
Diesel generator sets
Each server and each switch is connected to two power inputs. As a rule, in modern equipment such an opportunity is provided by the manufacturer. For servers with one input, the power is duplicated using this device:
Static load switch + manual bypass Digital Energy
Cooling system
For a comfortable life of the equipment in the machine rooms you need to maintain a certain temperature range. That is why companies around the world are increasingly building their data centers somewhere near the Arctic Circle - in such conditions you can use the outside air to cool the servers. This is called “free cooling”, and this approach is considered to be the most energy efficient (why waste energy on cooling warm air, if you can take the cold air right away?).
We also use freecooling, but with some reservations. Despite the legendary Petersburg coolness, in summer the air temperature still sometimes rises above the cherished 20-25 °, and it has to be further cooled. In winter, by contrast, the air is too cold to be used immediately. In addition to the fact that servers can simply overcool, a change in temperature will lead to a shift in the dew point - and condensate is contraindicated in such equipment. In addition, the air with such a scheme gets from the street, which means that it must be further cleaned.
Free cooling is used in one of the four machine rooms; in the rest, the cooling system operates according to the classical scheme using precision air conditioners.
Indoor units of precision air conditioners
External units of precision air conditioners
Cold air from the mixing chamber or air conditioner is fed through a raised floor or air duct into the so-called “cold corridor”. This is an isolated space between the front sides of two rows of racks. Here it is:
Cold hall of the machine room
On the reverse side, the exhaust heated air enters the “hot corridor” - and from there it is sent back for refrigeration by freon to the indoor units of air conditioners. Thus, the circulation of clean (dust-free) air in the engine room is achieved.
Hot hall of the machine room
Structured Cabling System
Kilometers carefully laid wires. No words are needed here.
Fire extinguishing system
In our data center gas fire extinguishing system is provided. Actually gas (freon) is stored under pressure in cylinders. In the event of a fire on the signal from the sensor in the engine room, the valve will be activated, and the gas through the pipes will flow to the source of ignition.
Cylinders with freon
Manometer
Monitoring
All health indicators of the data center are monitored in real time. These are temperature (from sensors of equipment and indoors), power supply, load on network equipment - the data is displayed on the attendant displays and controlled by automation. If something went wrong, the monitoring system itself will send a message about the problem to the engineers (VKontakte and SMS).
Power supply monitoring
Cooling system monitoring
Access control system and security
Only employees can enter the territory; any door is equipped with an electronic lock with an access card reader. ITSVA is guarded on a 24/7 basis, video surveillance is conducted in each room.
Let's sum up
ITSVA has a very good location: just a few kilometers from the hometown of VKontakte and near a reliable source of electricity.
Here, the process of upgrading equipment and increasing energy efficiency continues. PUE (Power Usage Effectiveness), or the coefficient of energy efficiency is a key indicator of data center assessment. It is considered as the ratio of all energy consumed by the data center, and the actual consumption of servers and network equipment. As is clear from this definition, the PUE of an ideal data center in a vacuum is 1.0. ITSVA is not an ideal data center in a vacuum, but we are systematically working on reducing this indicator.
The team of ICVA staff and network engineers on VKontakte is doing everything so that you can enjoy your favorite videos every day, watch new photos of friends and not worry about how complex infrastructure is used.
If you are a first-class specialist, you have a chance to get acquainted with our data center, becoming part of the team. We invite to work the head of the test lab of the data center and system administrators .
PS Articles on the technical side of VKontakte can also be read in our blog in Russian and English. You can ask the author in the official community .
Source: https://habr.com/ru/post/332430/
All Articles