Cycle of articles "NetRack Guards: guarding the client IT infrastructure": the story of a man who knows everything
Have you ever wondered how your service company works? Have you ever thought about those who provide the permanent work of infrastructure and services from a “physical” point of view? We decided to share our experience in this matter and talk about how and at the expense of what we ensure the continuity and quality of our services. Our company provides a wide range of services from the rental and placement of equipment in the data center to Internet access ports. NetRack exists in the Russian market since 2011 and serves HighLoad customers: banks, media, TV companies, online gaming companies, payment systems. Our partners are the most reliable data centers in Moscow: StoreData , IXcellerate , SafeData , MMTS-9 . Using the example of one of them, the StoreData data center, we will tell you how, and by whose efforts, the continuity of the work of the IT infrastructure we provide for our customers is achieved.
The first hero of our historical cycle about the behind-the-scenes life of NetRack service company and its data center is a person who always keeps abreast, and the phone is always on. A person who knows everything about each element of the engineering systems of his data center and how to make everything work without failures.
The duty service never sleeps. The duty service operates in data center mode: 24 hours a day, seven days a week, 365 days a year. They have a job like this: monitor each of the nine large monitors and several small ones that hang around the perimeter of their small room. It would seem that not the most difficult job for young people who are still students or just graduated from high school. However, the duty service is the link between the engineers working in the data center, and the companies placing their equipment in the data center.
')
- San, you finally checked ups? - the thin blond leans back in his chair, meeting a colleague who has just returned from the next round. Literally shortly before, one of the monitors showed an unstable input voltage in uninterruptible power supplies. By the arrival of the authorities would not want to report accidents in his shift.
- Yes, everything seems to be fine, but I will still convey to Konstantin anyway ...
- Tell me what? - There was a low voice. - Good morning, guys!
A man of strong build, taller than average, in a bright plaid shirt and jeans comes into the room of the duty service with a kind smile. While the duty officer Alexander tells the person about the recent surge, the smile gives way to a serious expression. Mysterious Konstantin exhales, obviously noting something to himself.
- Ok, I'll take a look. Surely nothing terrible, but it is good that he said - the smile comes back again, Konstantin is sent to the first module of the data center StoreData.
A man in a bright plaid shirt and jeans - Konstantin Eremin, chief engineer of StoreData. His working day starts at nine in the morning, and first of all he receives a report from the duty service on the status of the data center equipment during its non-working hours. This is one of the main duties of the chief engineer: to know what is happening in the data center, even if he is on vacation, ill, or is out of the office. And, of course, the chief engineer is responsible for ensuring that the data center runs smoothly. Indeed, the uninterrupted operation of companies that rent equipment in it depends on the uninterrupted operation of the data center.
After talking with the on-duty service and reviewing their records, to finally make sure that, apart from a power surge, nothing else happened, the chief engineer should check the operation of the equipment of each module. Konstantin performs such a detour twice a day: in the morning and in the evening. The first thing to be checked is temperature. In the hot corridor - about plus 28 ° C, in the cold - 18-20 ° C. In addition, it is imperative to check the level of humidity in the room. Next - monitoring MSB. Indication of inputs shows their presence and operability, network input parameters are normal - Konstantin nods approvingly and continues to inspect the equipment.
The next stage is the electrical part. All switches are raised, automatic equipment is working, loads are checked. There are two city inputs, which means: the data center operates in a regular mode from the city power grid.
Next - check UPS. First of all, Konstantin looks at the entries in the menu on the display: “Exit”, “Load”, “System”, “Batteries” and “Log”. Each menu has its own data and its own settings. For example, in the "Journal" recorded all abnormal situations. "Here is a power surge," says the chief engineer for himself. Indeed, nothing terrible happened: just at some point the mains voltage went beyond its parameters, and the duty officer received a signal about the instability of the input voltage. However, employees of the duty office have been working in the data center for several years now and are quietly acting within the framework of the developed regulations.
For example, in the event of an emergency situation, the duty officers first of all report it to their superior and Konstantin. The telephone number of the chief engineer is always on - Konstantin is always in touch, because they can call him at any time, even at night. Situations are different: temperature increase; non-synchronization of inputs or minor deviations from the network’s reference parameters are also affected by work at MOESK (Moscow United Electric Grid Company) when, for example, equipment is switched or repaired there.
In the module Konstantin it remains only to check the battery charge and load, after which he makes sure that the fire-fighting equipment and the gas extinguishing system work properly. It is worth noting a special system of early warning and fire detection: air is constantly taken for analysis for the presence of products of combustion and smoke. In case of their detection, the sensor in the gas analyzer system detects smoke even at the earliest stages, which immediately notifies the duty service. Next, the chief engineer goes along the technical corridor of the "sarcophagus" - as Eremin calls the module for its second roof and walls. It remains to check the pressure level in gas fire extinguishing cylinders, and then go through the same circle to the second module - StoreData_Eco. Konstantin performs the same manipulations, but faster: in the second module of equipment and the load is less, so the verification process does not take much time.
The third in line for the morning round is StoreData_lite data center, which is still preparing to accept new NetRack customers. Here Konstantin practically does not linger, since the module works in test mode, then it descends to diesel generators.
On the inside of the StoreData data center there are two diesel generators: one with a capacity of 1,250 kW, the second with 450 kW. Konstantin examines them with special care and awe, as if it were his own cars in his personal garage.
He checks the motor for external faults, the status of internal monitoring parameters. A separate story - the temperature. It must be at least 32 ° C, because they are in hot reserve. In addition, the chief engineer checks the batteries and the presence of fuel in the tank. He pays special attention to the DGU cleanliness: there should not be any smudges, lubricant or antifreeze, because any excess drop can affect the operation of the entire generator. The last glance is at the monitor controlling the operation of the diesel generator: all parameters are normal, the yellow indicator is on, indicating that the diesel generator set is on and ready to go.
Both in the DGU and in each module there are logs with records of each round of the duty service. Since such detours are performed several times a day, Konstantin studies them carefully, and then goes to the office with a calm soul: the detour is over - you can do current affairs.
But such cases from the chief engineer enough. To begin with, he prefers to familiarize himself with the correspondence of the duty service with the clients in order to finally make sure that in his absence there really haven't been any abnormal situations. Sometimes Konstantin has to communicate with customers himself, but this happens only when client questions go beyond the boundaries of the competence of the specialists on duty. In general, communication with customers for him is limited to the first visit of the customer, when the chief engineer listens to all his wishes on the technical equipment, for example, how many sockets are needed, where the equipment should be located, etc.
The chief engineer is in closer contact with contractors who regularly carry out maintenance work on the maintenance of UPS, air-conditioning systems and power supply and diesel generator sets.
As Konstantin says, one of his tasks is constant modernization. This applies to all equipment in the data center, up to the complete replacement of fluorescent lighting with LED or replacement of all batteries due to wear.
“I also connect everyone together so that everything works as one mechanism,” said Konstantin. He has in mind his subordinates: engineers of a narrower profile and employees of contracting services who carry out maintenance and maintenance work. Konstantin does not have his own office: he sits in a room with other engineers and a technical director, the process of interaction and control takes place continuously. On the far wall hangs a large plasma panel that transmits general data monitoring modules of the data center, so that all interested parties are aware of what is happening in the data center. The situation is sometimes noisy, and sometimes, on the contrary, too calm: either everyone is talking on the phone about something, or they are doing some kind of paper work. However, Konstantin does not consider this a routine: as he himself says, the main thing is that you yourself observe everything, and where necessary, you can work with your hands.
While the essence of the matter, it is time for lunch - a time when you can relax a bit and discuss things not related to work. Konstantin, for example, shares his impressions of his last visit to Tsarsky Apiary in Izmailovsky Park. Yes, the chief engineer of the company StoreData grew up in a family of builders, but he spent part of his childhood in the apiary, in Altai, and in the capital he found a place where he could return to his childhood hobby. Izmailovsky Park not only provides an opportunity to remember childhood, but also to consolidate the knowledge that Konstantin once received: on the territory of this apiary, he and the refrigeration engineer Yevgeny are caring for bees.
But the work does not wait: it is time for an evening round. It is no different from the morning, except that if some kind of error occurs or the equipment fails, the chief engineer will have to stay until the problem is fixed, even if it is much later than the end of the working day. By the way, such detours are not the only case when Konstantin checks the stability of equipment operation. Every two weeks, a scheduled check of diesel generators is carried out at idle: output parameters, output voltage, fuel level and engine condition are checked. In addition, all data center systems are tested twice a year. An emergency situation is simulated: engineers imitate the disappearance of one of the inputs and both at once, that is, complete disconnection from the city power supply, and watch how the equipment “runs out”. Customers are always warned about this in advance, according to the regulations. However, the topology of the engineering infrastructure of the data center allows for such work without stopping the provision of customer service - for two years of work by Konstantin Eremin at StoreData there were no problems that led to accidents in the data center, and the customers during such checks have never complained about failures the work of the data center.
Konstantin is a little silent on questions about personal time and hobbies: he remembers only not without nostalgia about childhood, when he was a member of the junior national team of Kazakhstan in handball. “But then there was an injury that gently hinted to me that it’s time to finish with the sport,” with a smile, albeit a little sad, says the chief engineer.
After the detour, Konstantin will once again check the duty service logs, make sure that all the systems are in order and all the necessary work is done, then go home so that the next day again at nine in the morning with a smile, greet the guys from the duty service and go to inspect the data center. Indeed, only under the careful and constant monitoring of the chief engineer, the data center equipment will work without failures 24 hours a day, seven days a week, 365 days a year. In the meantime, the data center StoreData works, users of NetRack receive uninterrupted quality of service of the services provided.
The first hero of our historical cycle about the behind-the-scenes life of NetRack service company and its data center is a person who always keeps abreast, and the phone is always on. A person who knows everything about each element of the engineering systems of his data center and how to make everything work without failures.
The duty service never sleeps. The duty service operates in data center mode: 24 hours a day, seven days a week, 365 days a year. They have a job like this: monitor each of the nine large monitors and several small ones that hang around the perimeter of their small room. It would seem that not the most difficult job for young people who are still students or just graduated from high school. However, the duty service is the link between the engineers working in the data center, and the companies placing their equipment in the data center.
')
Data Center StoreData, 09:00
- San, you finally checked ups? - the thin blond leans back in his chair, meeting a colleague who has just returned from the next round. Literally shortly before, one of the monitors showed an unstable input voltage in uninterruptible power supplies. By the arrival of the authorities would not want to report accidents in his shift.
- Yes, everything seems to be fine, but I will still convey to Konstantin anyway ...
- Tell me what? - There was a low voice. - Good morning, guys!
A man of strong build, taller than average, in a bright plaid shirt and jeans comes into the room of the duty service with a kind smile. While the duty officer Alexander tells the person about the recent surge, the smile gives way to a serious expression. Mysterious Konstantin exhales, obviously noting something to himself.
- Ok, I'll take a look. Surely nothing terrible, but it is good that he said - the smile comes back again, Konstantin is sent to the first module of the data center StoreData.
A man in a bright plaid shirt and jeans - Konstantin Eremin, chief engineer of StoreData. His working day starts at nine in the morning, and first of all he receives a report from the duty service on the status of the data center equipment during its non-working hours. This is one of the main duties of the chief engineer: to know what is happening in the data center, even if he is on vacation, ill, or is out of the office. And, of course, the chief engineer is responsible for ensuring that the data center runs smoothly. Indeed, the uninterrupted operation of companies that rent equipment in it depends on the uninterrupted operation of the data center.
After talking with the on-duty service and reviewing their records, to finally make sure that, apart from a power surge, nothing else happened, the chief engineer should check the operation of the equipment of each module. Konstantin performs such a detour twice a day: in the morning and in the evening. The first thing to be checked is temperature. In the hot corridor - about plus 28 ° C, in the cold - 18-20 ° C. In addition, it is imperative to check the level of humidity in the room. Next - monitoring MSB. Indication of inputs shows their presence and operability, network input parameters are normal - Konstantin nods approvingly and continues to inspect the equipment.
The next stage is the electrical part. All switches are raised, automatic equipment is working, loads are checked. There are two city inputs, which means: the data center operates in a regular mode from the city power grid.
Next - check UPS. First of all, Konstantin looks at the entries in the menu on the display: “Exit”, “Load”, “System”, “Batteries” and “Log”. Each menu has its own data and its own settings. For example, in the "Journal" recorded all abnormal situations. "Here is a power surge," says the chief engineer for himself. Indeed, nothing terrible happened: just at some point the mains voltage went beyond its parameters, and the duty officer received a signal about the instability of the input voltage. However, employees of the duty office have been working in the data center for several years now and are quietly acting within the framework of the developed regulations.
For example, in the event of an emergency situation, the duty officers first of all report it to their superior and Konstantin. The telephone number of the chief engineer is always on - Konstantin is always in touch, because they can call him at any time, even at night. Situations are different: temperature increase; non-synchronization of inputs or minor deviations from the network’s reference parameters are also affected by work at MOESK (Moscow United Electric Grid Company) when, for example, equipment is switched or repaired there.
In the module Konstantin it remains only to check the battery charge and load, after which he makes sure that the fire-fighting equipment and the gas extinguishing system work properly. It is worth noting a special system of early warning and fire detection: air is constantly taken for analysis for the presence of products of combustion and smoke. In case of their detection, the sensor in the gas analyzer system detects smoke even at the earliest stages, which immediately notifies the duty service. Next, the chief engineer goes along the technical corridor of the "sarcophagus" - as Eremin calls the module for its second roof and walls. It remains to check the pressure level in gas fire extinguishing cylinders, and then go through the same circle to the second module - StoreData_Eco. Konstantin performs the same manipulations, but faster: in the second module of equipment and the load is less, so the verification process does not take much time.
The third in line for the morning round is StoreData_lite data center, which is still preparing to accept new NetRack customers. Here Konstantin practically does not linger, since the module works in test mode, then it descends to diesel generators.
On the inside of the StoreData data center there are two diesel generators: one with a capacity of 1,250 kW, the second with 450 kW. Konstantin examines them with special care and awe, as if it were his own cars in his personal garage.
He checks the motor for external faults, the status of internal monitoring parameters. A separate story - the temperature. It must be at least 32 ° C, because they are in hot reserve. In addition, the chief engineer checks the batteries and the presence of fuel in the tank. He pays special attention to the DGU cleanliness: there should not be any smudges, lubricant or antifreeze, because any excess drop can affect the operation of the entire generator. The last glance is at the monitor controlling the operation of the diesel generator: all parameters are normal, the yellow indicator is on, indicating that the diesel generator set is on and ready to go.
Both in the DGU and in each module there are logs with records of each round of the duty service. Since such detours are performed several times a day, Konstantin studies them carefully, and then goes to the office with a calm soul: the detour is over - you can do current affairs.
But such cases from the chief engineer enough. To begin with, he prefers to familiarize himself with the correspondence of the duty service with the clients in order to finally make sure that in his absence there really haven't been any abnormal situations. Sometimes Konstantin has to communicate with customers himself, but this happens only when client questions go beyond the boundaries of the competence of the specialists on duty. In general, communication with customers for him is limited to the first visit of the customer, when the chief engineer listens to all his wishes on the technical equipment, for example, how many sockets are needed, where the equipment should be located, etc.
The chief engineer is in closer contact with contractors who regularly carry out maintenance work on the maintenance of UPS, air-conditioning systems and power supply and diesel generator sets.
As Konstantin says, one of his tasks is constant modernization. This applies to all equipment in the data center, up to the complete replacement of fluorescent lighting with LED or replacement of all batteries due to wear.
“I also connect everyone together so that everything works as one mechanism,” said Konstantin. He has in mind his subordinates: engineers of a narrower profile and employees of contracting services who carry out maintenance and maintenance work. Konstantin does not have his own office: he sits in a room with other engineers and a technical director, the process of interaction and control takes place continuously. On the far wall hangs a large plasma panel that transmits general data monitoring modules of the data center, so that all interested parties are aware of what is happening in the data center. The situation is sometimes noisy, and sometimes, on the contrary, too calm: either everyone is talking on the phone about something, or they are doing some kind of paper work. However, Konstantin does not consider this a routine: as he himself says, the main thing is that you yourself observe everything, and where necessary, you can work with your hands.
While the essence of the matter, it is time for lunch - a time when you can relax a bit and discuss things not related to work. Konstantin, for example, shares his impressions of his last visit to Tsarsky Apiary in Izmailovsky Park. Yes, the chief engineer of the company StoreData grew up in a family of builders, but he spent part of his childhood in the apiary, in Altai, and in the capital he found a place where he could return to his childhood hobby. Izmailovsky Park not only provides an opportunity to remember childhood, but also to consolidate the knowledge that Konstantin once received: on the territory of this apiary, he and the refrigeration engineer Yevgeny are caring for bees.
But the work does not wait: it is time for an evening round. It is no different from the morning, except that if some kind of error occurs or the equipment fails, the chief engineer will have to stay until the problem is fixed, even if it is much later than the end of the working day. By the way, such detours are not the only case when Konstantin checks the stability of equipment operation. Every two weeks, a scheduled check of diesel generators is carried out at idle: output parameters, output voltage, fuel level and engine condition are checked. In addition, all data center systems are tested twice a year. An emergency situation is simulated: engineers imitate the disappearance of one of the inputs and both at once, that is, complete disconnection from the city power supply, and watch how the equipment “runs out”. Customers are always warned about this in advance, according to the regulations. However, the topology of the engineering infrastructure of the data center allows for such work without stopping the provision of customer service - for two years of work by Konstantin Eremin at StoreData there were no problems that led to accidents in the data center, and the customers during such checks have never complained about failures the work of the data center.
Konstantin is a little silent on questions about personal time and hobbies: he remembers only not without nostalgia about childhood, when he was a member of the junior national team of Kazakhstan in handball. “But then there was an injury that gently hinted to me that it’s time to finish with the sport,” with a smile, albeit a little sad, says the chief engineer.
After the detour, Konstantin will once again check the duty service logs, make sure that all the systems are in order and all the necessary work is done, then go home so that the next day again at nine in the morning with a smile, greet the guys from the duty service and go to inspect the data center. Indeed, only under the careful and constant monitoring of the chief engineer, the data center equipment will work without failures 24 hours a day, seven days a week, 365 days a year. In the meantime, the data center StoreData works, users of NetRack receive uninterrupted quality of service of the services provided.
Source: https://habr.com/ru/post/310060/
All Articles