Engineering systems of our data centers and their monitoring, part one
Hi, Habr! I am an engineer at Miran, which leases various servers, accommodates client equipment in its data centers, and other similar matters.
For a long time I was persuaded to write an overview article about our engineering infrastructure and how we monitor it. The article did not want to be born long and hard, but it was still born. Congratulate me with the initiative!
')
Careful, a lot of pictures!
1. Introduction to Chain Management
2. TP, I LIE, PDU
3. Immerse yourself in the coolness ©
4. From terminal clamp to PLC
What is a data center? We say "data center" - imagine the endless hangars, filled with countless rows of racks with a steady buzzing glands. Pieces mysteriously wink at the multi-colored lights from the twilight. From the front they are blown with a cool breeze from industrial air conditioners. In cold zones, you can slip your neck and catch a cold. Therefore, admins always wear sweaters.
Our company has built two separate data centers, unsophisticatedly referred to as Miran-1 and Miran-2. The first is a completely familiar type, with one large machine room and several smaller floors above. The second data center is a hangar, which currently has two mobile small data centers installed, as well as a third one. Mobile data centers - two-storey structures-containers, the first floor of which is a server room with racks and air conditioners, it is also called a server unit, on the second floor I LIE installed, installed UPS and various control panels.
Historically, Miran-1 did not have a single monitoring of the engineering infrastructure (we sprinkle ashes on our heads) and we are trying to correct this deficiency. Therefore, it is mostly about the second data center.
All pictures are taken from our monitoring system!
PS: photos of shields and their entrails are also ours!
Miran-2 has a guaranteed power supply system (SGE). As can be seen from the diagram below, under normal conditions the data center is powered by two independent external inputs from the TP ; in case of power failure at external inputs (and this sometimes happens with us) - the power comes from the diesel generator set DGU2, in fact; Under the future reserve space is provided for two more.
General miranosmir power input "Miran-2"
Go ahead. I LIE made two-section with a sectional switch under the control of AVR1. The AVR controller will close the slacker in case of power failure on one or both inputs, in the latter case, after 15 seconds, a signal will be given to start the diesel generator set. All these troubles "Module-1 and" Module-2 "are experiencing on their internal UPS.
Photos of the VRU "Miran-2". Clickable
The main purpose of the sections and their machines, in addition to powering various auxiliary lighting panels, ventilation control, and other things, is to play the role of power supply inputs for Module-1 and Module-2 (QF1.1-.2 and QF2.1-.2 on scheme, respectively). Each modular data center has its own IGU inside.
Mnemonic scheme of the main switchboard "Module-2"
Photo of the main switchboard "Module-2". Clickable
Mnemonic of the “Module-1” power unit
Mnemoscheme racks "Module-1"
Most of the racks in Modul-1 and Modul-2 are from Rittal and RiT. From the PDU we use: in Module 1, a hodgepodge from Eurolan, APC, DELTA. Module-2 is entirely on the RiT PDU.
All client hardware, as well as the engineering infrastructure in the course of their work, emit a lot of heat. This heat must be removed, otherwise the iron will die quickly. We deal with the removal of six inverter freon air conditioners company Daikin. All their activities are proudly called “freon mode”, which provides a drytropical tropical climate from +15 to +23 ° in a cold corridor. This cooling system is used both in Modula-1 and Module-2.
Also in Modula-1, there is another cooling mode, the “free cooling mode”. It should be provided with four air handling units and a dozen extracts. In theory. Unfortunately, in practice, the heat removal in this way was not very efficient if a little more than half of the racks were used inside. Therefore, this mode is not used for the first modular data center, remaining, in fact, back-up.
Mnemonic of the server module "Module-1"
The mnemonic diagram of the module-2 server block. No blowers, justhardcore! freon!
Three PLCs are engaged in polling and aggregation of information from the entire periphery of the Miran-2 data center: one for the Module and one for the total. These little trunks are named after the well-known company WAGO.
Consider the structure of the polling system based on the solution for “Module-2”.
PLC bus diagram with modules, screenshot from WAGO-IO-Check
Photo dispatch board "Module-2". Clickable
As can be seen from the diagram, the PLC of the 750-881 series is installed on the bus, four discrete modules 750-1405 for 16 channels each and one analog module 750-455 for four channels. Through discrete modules, the PLC receives data on the state of the automatic power switches (“dry” auxiliary contacts) in both sections of the main switchboard , on the state of the circuit breakers in its own switchboard , and on the ventilation status of the power unit. Through the analog module - it receives data from two temperature and humidity sensors (4-20 mA) here, inside the power unit.
The PLC is also equipped with two Ethernet ports and through them it communicates via Modbus TCP / IP with several more hardware, such as:
Photo ABB CMS-600 and current transformers. Clickable
Photo OWEN ME110-220.3M. Clickable
Photo of CAREL PLC. Clickable
Photograph of a shield of low-current Modul-2 systems. Clickable
Photo UPS "Module-2". Clickable
All the information received through the program is formed into its own list of Modbus registers, which is already polled by SCADA .
The second figure, soft
For a long time I was persuaded to write an overview article about our engineering infrastructure and how we monitor it. The article did not want to be born long and hard, but it was still born. Congratulate me with the initiative!
')
Careful, a lot of pictures!
1. Introduction to Chain Management
2. TP, I LIE, PDU
3. Immerse yourself in the coolness ©
4. From terminal clamp to PLC
Introduction to Chain Management
What is a data center? We say "data center" - imagine the endless hangars, filled with countless rows of racks with a steady buzzing glands. Pieces mysteriously wink at the multi-colored lights from the twilight. From the front they are blown with a cool breeze from industrial air conditioners. In cold zones, you can slip your neck and catch a cold. Therefore, admins always wear sweaters.
Our company has built two separate data centers, unsophisticatedly referred to as Miran-1 and Miran-2. The first is a completely familiar type, with one large machine room and several smaller floors above. The second data center is a hangar, which currently has two mobile small data centers installed, as well as a third one. Mobile data centers - two-storey structures-containers, the first floor of which is a server room with racks and air conditioners, it is also called a server unit, on the second floor I LIE installed, installed UPS and various control panels.
Historically, Miran-1 did not have a single monitoring of the engineering infrastructure (we sprinkle ashes on our heads) and we are trying to correct this deficiency. Therefore, it is mostly about the second data center.
All pictures are taken from our monitoring system!
PS: photos of shields and their entrails are also ours!
TP, I LIE, PDU
Miran-2 has a guaranteed power supply system (SGE). As can be seen from the diagram below, under normal conditions the data center is powered by two independent external inputs from the TP ; in case of power failure at external inputs (and this sometimes happens with us) - the power comes from the diesel generator set DGU2, in fact; Under the future reserve space is provided for two more.
General miranosmir power input "Miran-2"
Go ahead. I LIE made two-section with a sectional switch under the control of AVR1. The AVR controller will close the slacker in case of power failure on one or both inputs, in the latter case, after 15 seconds, a signal will be given to start the diesel generator set. All these troubles "Module-1 and" Module-2 "are experiencing on their internal UPS.
Photos of the VRU "Miran-2". Clickable
The main purpose of the sections and their machines, in addition to powering various auxiliary lighting panels, ventilation control, and other things, is to play the role of power supply inputs for Module-1 and Module-2 (QF1.1-.2 and QF2.1-.2 on scheme, respectively). Each modular data center has its own IGU inside.
Mnemonic scheme of the main switchboard "Module-2"
Photo of the main switchboard "Module-2". Clickable
Mnemonic of the “Module-1” power unit
Mnemoscheme racks "Module-1"
Most of the racks in Modul-1 and Modul-2 are from Rittal and RiT. From the PDU we use: in Module 1, a hodgepodge from Eurolan, APC, DELTA. Module-2 is entirely on the RiT PDU.
Plunge into the coolness ©
All client hardware, as well as the engineering infrastructure in the course of their work, emit a lot of heat. This heat must be removed, otherwise the iron will die quickly. We deal with the removal of six inverter freon air conditioners company Daikin. All their activities are proudly called “freon mode”, which provides a dry
Also in Modula-1, there is another cooling mode, the “free cooling mode”. It should be provided with four air handling units and a dozen extracts. In theory. Unfortunately, in practice, the heat removal in this way was not very efficient if a little more than half of the racks were used inside. Therefore, this mode is not used for the first modular data center, remaining, in fact, back-up.
Mnemonic of the server module "Module-1"
The mnemonic diagram of the module-2 server block. No blowers, just
From terminal clamp to PLC
Three PLCs are engaged in polling and aggregation of information from the entire periphery of the Miran-2 data center: one for the Module and one for the total. These little trunks are named after the well-known company WAGO.
Consider the structure of the polling system based on the solution for “Module-2”.
PLC bus diagram with modules, screenshot from WAGO-IO-Check
Photo dispatch board "Module-2". Clickable
As can be seen from the diagram, the PLC of the 750-881 series is installed on the bus, four discrete modules 750-1405 for 16 channels each and one analog module 750-455 for four channels. Through discrete modules, the PLC receives data on the state of the automatic power switches (“dry” auxiliary contacts) in both sections of the main switchboard , on the state of the circuit breakers in its own switchboard , and on the ventilation status of the power unit. Through the analog module - it receives data from two temperature and humidity sensors (4-20 mA) here, inside the power unit.
The PLC is also equipped with two Ethernet ports and through them it communicates via Modbus TCP / IP with several more hardware, such as:
- two Schneider Electric input automata, which also provide information on input capacities, voltages, currents, and so on;
- two ABB current measurement systems in tandem with two input modules from the firm OWEN - the result of their joint work is the calculation of post-rack power;
- CAREL controllers and their 6 wards - Daikin air conditioners;
- and finally, the younger brother - a 750-342 cpl with seven discrete modules. Their task is to monitor the status of 48 switches + 12 reserve ones in the server block for 24 racks.
Photo ABB CMS-600 and current transformers. Clickable
Photo OWEN ME110-220.3M. Clickable
Photo of CAREL PLC. Clickable
Photograph of a shield of low-current Modul-2 systems. Clickable
Separately, it is worth mentioning the UPS, they are polled directly by SCADA, bypassing the PLC, via the SNMP protocol.
Photo UPS "Module-2". Clickable
All the information received through the program is formed into its own list of Modbus registers, which is already polled by SCADA .
A small piece of the main program
(* PLC_A2 *) %QX256.0 := A2_1QF1; // 256 %QX256.1 := A2_1QF2; // %QX256.2 := A2_QS1; %QX256.3 := A2_QS2; %QX256.4 := A2_3QF1; %QX256.5 := A2_3QF2; %QX256.6 := A2_3QF3; %QX256.7 := A2_3QF4; %QX256.8 := A2_3QF5; %QX256.9 := A2_3QF6; %QX256.10 := A2_3QF7; %QX256.11 := A2_3QF8; %QX256.12 := A2_3QF9; %QX256.13 := A2_3QF10; %QX256.14 := A2_KM1; %QX256.15 := A2_KM2; (* QF1 *) // â„– 1 %QW332 := QF1_I_L1; // %QW333 := QF1_I_L2; %QW334 := QF1_I_L3; %QW335 := QF1_U_L12; // () %QW336 := QF1_U_L23; %QW337 := QF1_U_L31; %QW338 := QF1_U_L1; // (-) %QW339 := QF1_U_L2; %QW340 := QF1_U_L3; %QW341 := QF1_P_L1; // %QW342 := QF1_P_L2; %QW343 := QF1_P_L3; %QW344 := QF1_P_Sum; // () %QW345 := QF1_Q_L1; // %QW346 := QF1_Q_L2; %QW347 := QF1_Q_L3; %QW348 := QF1_Q_Sum; // () %QW349 := QF1_S_Sum; // () %QW350 := QF1_CosF; // Another piece from another subroutine
// CODESYS, // Modbus TCP/IP. , , // 110-220.3 // - PROGRAM MBCFG_subCMS_1(* generated by config one prg for each slave *) VAR_OUTPUT U_L1 : WORD; (**) U_L2 : WORD; (**) U_L3 : WORD; (**) /*--- system variables (read only) ----------------------------------------*/ MBCFG_IpAddress : STRING(12) := 'XXX.XXX.XXX.XXX';//IP- Slave- MBCFG_Port : UINT := 502; //, MBCFG_UnitID : BYTE := 2; //ID Slave- MBCFG_TimeOut : TIME := t#300ms; // MBCFG_RequestDelay : TIME := t#1000ms; // MBCFG_Error : MBCFG_eERROR := MBCFG_START_UP; MBCFG_LastJob : MBCFG_typCOM_JOB; /*-------------------------------------------------------------------------*/ END_VAR VAR CONSTANT zz_VARIABLECOUNT: INT := 3; (* number of variables *) zz_JOBCOUNT : INT := 1; (* number of jobs *) END_VAR VAR /*=== VARIABLE LIST =============*/ zz_VariableList : ARRAY[1..zz_VARIABLECOUNT] OF MBCFG_typVARIABLE := ( DataType := MBCFG_TYPE_WORD, ByteOrder := MBCFG_BYTE_ORDER_0, BitSize := 16, ptVar := 0, ReadJobIndex := 1, ReadStartBitNo := 0, WriteJobIndex := 0, WriteStartBitNo := 0 ), ( DataType := MBCFG_TYPE_WORD, ByteOrder := MBCFG_BYTE_ORDER_0, BitSize := 16, ptVar := 0, ReadJobIndex := 1, ReadStartBitNo := 32, WriteJobIndex := 0, WriteStartBitNo := 0 ), ( DataType := MBCFG_TYPE_WORD, ByteOrder := MBCFG_BYTE_ORDER_0, BitSize := 16, ptVar := 0, ReadJobIndex := 1, ReadStartBitNo := 64, WriteJobIndex := 0, WriteStartBitNo := 0 ); /*=== JOB LIST ==================*/ zz_JobList : ARRAY[1..zz_JOBCOUNT] OF MBCFG_typCOM_JOB := ( Functioncode := 3, // , 0x03, Read Holding Registers ReadStartAddress := 26,// ReadQuantity := 5, //- , WriteStartAddress := 0, WriteQuantity := 0, ptReadData := 0, ptWriteData := 0 ); zz_DataField_1_Read : ARRAY[1..5] OF WORD; /*=== MODBUS MASTER ==============*/ zz_MBCFG_MASTER_ETH : MBCFG_MASTER_TCP; END_VAR /*--- for each variable -------------------------*/ zz_VariableList[1].ptVar := ADR(U_L1); zz_VariableList[2].ptVar := ADR(U_L2); zz_VariableList[3].ptVar := ADR(U_L3); /*-----------------------------------------------*/ /*--- for each job -----------------------------------*/ zz_JobList[1].ptReadData := ADR(zz_DataField_1_Read); /*----------------------------------------------------*/ /*#### START OF FIXED CODE #####################################*/ zz_MBCFG_MASTER_ETH( strIpAddress := MBCFG_IpAddress, uiPort := MBCFG_Port, bUnitID := MBCFG_UnitID, tTimeOut := MBCFG_TimeOut, iVariableCount := zz_VARIABLECOUNT, ptVariableList := ADR(zz_VariableList), iJobCount := zz_JOBCOUNT, ptJobList := ADR(zz_JobList), tRequestDelay := MBCFG_RequestDelay, eError => MBCFG_Error, LastJob => MBCFG_LastJob ); %QW377 := U_L1; %QW378 := U_L2; %QW379 := U_L3; The second figure, soft
Source: https://habr.com/ru/post/336134/
All Articles