The work of an “IT astronaut”: how we engineers go on a visit to bears and seals

"Heck. Launch the cosmonaut ”- a phrase with which our most difficult trips begin. I remember last Christmas, something like this, and we went to repair ABKHM of a public institution. The machine froze, and the data center warmed up from it. It depended on whether we repaired or not how older people throughout the country would spend the New Year (with or without a pension). Saved, but the work was still that. Then it was calculated and accrued. Retirement

Since then, we have been called astronauts, because we work in autonomous system. At least half of trips is a maximum of an hour for a complete set and a week somewhere where instead of Habr only bears. And even money doesn't cost anything.
')

About astronauts

It so happened that now the whole of KROK knows us as an astronaut detachment. We have two “special forces” visiting teams for particularly difficult tasks - colleagues on heavy servers have an “X-team”, and we have “cosmonauts”. The X-team travels more in Moscow and in the center of the country - yet heavy iron here is stumbled apart from more than the Arctic Circle. Yes, and remote control almost always solves, unless you configure firewalls. Network salespeople generally managed to somehow coordinate the operation of connecting approximately 50 points beyond the Urals at the enterprises by local engineers. And there is such specificity - it was necessary to make a gap at the entrance to the Internet channel and connect a new piece of metal. As soon as the gap appeared - even telephony does not work. Therefore, if you did not write the instructions in plain simple language - you will go to visit the bears. Then it happened.

And we often travel for preventive maintenance, various repairs of server, data center engineers and various related infrastructure items. Refrigerators and UPS systems cannot be worked off remotely, so everyone is at least 4-5 cities in a month. The country knows, like its own five fingers, the features of airports, canteens for staff, all sorts of nychkas, the locations of charging sockets in various public buildings, too.

About my work

I'll start with myself and how the work is generally arranged. Colleagues it is arranged in the same way, maybe the details are different.

My name is Roman Karpov. I have a profile education - an electrician for commercial and refrigeration equipment, the second higher education is a leading engineer for ventilation and air conditioning systems. In the team 5 years. He started as a technician, before that he worked in the protection of Rosatom (more precisely in the private security company). In the army he served as commander of the fire brigade, planned to go to the military school of the Emergencies Ministry. At that time, there was no education at all, only a very loud checkmate and combat deployment. By the way, I still keep within the standard: with the attack staircase - to the 4th floor from 35 meters in 42 seconds.

I had a trial period of 2 months - I didn’t understand anything, I sat, read the documentation for the equipment, traveled with my senior comrades on trips around Moscow. The first combat independent departure was to St. Petersburg - it was necessary to make the data center service, there were difficulties with air conditioning. The thermostatic valve failed, a replacement was required, plus there was a set of works to put the air conditioning system into operation. Day four is gone. Everything was clear, understandable, the main thing was to take and make it by hand, to come to an agreement with people to help.

This was the last departure without surprises. The next 5 years were surprises every day. There are a lot of trips. For example, in 20 days I visited Rostov-on-Don, Chita, Ulan-Ude, Surgut, Naberezhnye Chelny. There was a couple of days off between Chita and Ulan-Ude, we went to Baikal with colleagues, we have long wanted. Then again arbeiten.


It was Pavel Stepanov and I who were recently driven off to Ulan-Ude. It is beautiful even on the road.

The specifics of the team - you need to be able to jump and run at any time. Constantly someone on duty. There are service contracts and accidents. Service contracts are when, in a limited time, N objects are to be driven around and maintenance, repairs, maintenance and so on are done there. We can do this, probably, the best in the country. Why and what is needed for this - a little later, when we talk about Hot Key and the delivery of materials there. In the meantime, I will tell about the application.

So, I, as a leading engineer of the direction, make up a schedule of visits, bring it to the team and plan my time. For example, 20 days of the month - prevention in the country (November 2–7 - Chita and Ulan-Ude; November 9–13 - Surgut; 16–17 - Rostov-on-Don; 19–21 - Naberezhnye Chelny; 23–27 worked quietly in Moscow). I go home almost every weekend: business trips are short, usually 2-3 days (if there are airplanes within a radius of 400 kilometers from the facility). Colleagues have a similar schedule, the only time, for example, if a spouse is pregnant, then we try not to send a person further than Moscow region, and so on. Or when a person just returned from Perm, then according to a new application, the one who was in Moscow yesterday would go, and not he from the plane to the plane. Although, of course, sometimes it is necessary and so to do when accidents happen at the same time somewhere in different places.

Day

On duty, we are doing current office work and waiting for applications. We study the documentation, fiddle with reports, disassemble, assemble interesting nodes, solder rarely killed things, teach people. Statistically, accidents should be evenly distributed on schedule, but in practice this is far from the case. Usually breakdowns happen in pairs, and at once in two different parts of the country. And not on Monday morning, no. Bid Monday morning is a holiday. Usually, the alarm starts at 23:00 on Friday.

Next - an attempt to understand remotely what happened, whether you need to go. In the X-team and networkers frequent problem - power failure or problems in the highway, so they leave in the case in about a third of cases. We have 90% of accidents - the question is right for us, and that's all.

“Past” looks like this - on January 3, at 9 am, Pasha went to the site. Asks where the fault is. The guard shows to the monitor: behold, they say, it turned off after the alarm and does not turn on. Pasha moved his mouse - the monitor turned on. The guard continues: what is the password then? .. In general, it was not clear to them who to call, everyone was just mooing - so they set the ticket for us.

So, bid. The application is assigned responsible. For example me. If there is an SLA per day (most often there is), you need to see what's with the ticket. For the most distant and important flights, sometimes we keep tickets already purchased with an open date, but more often you just need to book. If the office is working, the resource allocation team can help; if not, the order is usually made already in a taxi to the airport. Another warehouse is an urgent one in our building, and a large one with rare things is in Khimki, a specialist from there can take what you need to the airport. But these are isolated cases, most often - 12 kilograms of tools, a laptop - and more.

If on-site operators do not run in circles shouting "everything is lost" (but only in one word) - we live through the equipment. We, in contrast, again, from the X-team, we can do anything. In the server park, the replacement is only with modules, and we solder, assemble one of the other (actually, we often disassemble non-critical systems and unconventionally use their components for repair in cities where parts cannot be reached quickly). Virtually none of the vendors (with two exceptions, in particular, when the customer bought a spare parts kit from the vendor) cannot dictate the conditions for repair and support - they are not removed from the guarantee for our actions. If we were waiting for specialists in engineering vendors, the plants in the country would stand.


Preserved photo from my first subject

My schedule is usually known (approximately, if there are no major accidents) for the week ahead - there is no point in planning further. It is possible, but, as experience shows, it is not necessary. You plan this for a couple of weeks - then something will happen. This is almost a sign, no one is planning to go further from the week. And still it is better to take return tickets on the last day - from the same opera.

The year is not painted, the quarter too. Heck. Right now, while I am writing, the application has come. In Nizhny Novgorod, the cooling system does not start. Thank you all, we went to work.

...

Something like that. Exit work - home, family, normally have time. No children yet. There will be - I will move to the office, drive around Moscow and St. Petersburg (there are such options).

Departures

Usually long trips are about a week. In Khabarovsk, I was with a colleague for two weeks. Difficult. Hung in the balance of the data center of a large cooling bank (in summer). There was a great risk that processes would arise. There are very difficult freon leaks from the circuit in place, it was necessary to look for and fix them. The story is this: the chiller was monoblock with an air-cooled condenser and a water-cooled evaporator with free cooling. 4 compressors, 2 circuits, in both 407th multicomponent. And this is such a freon that according to the norms it is refilled only with the whole volume, because otherwise the proportions of the components will be unbalanced. Process: the car stops, freon is evacuated from the system into the tank (it is impossible to get into the atmosphere). Then, using the method of dry nitrogen under overpressure, the system is filled to the recommended values, and work is carried out on finding leaks. We go, we look at the pipes - we carry out a detailed inspection of the nodes using special means. Leaks are determined. Then the nitrogen from the system is evacuated. Pipes are soldered. Now we need to check how the leaks were closed, that is, leak tightness according to the norms. Nitrogen pressure check again. It withstands the terms of the standard and there is no pressure drop - it means that it is tight. The act is drawn up and approved. Again, nitrogen bleed. Then the system is evacuated and the filling is done again with freon and compressor oil in case of volume loss. Then the cars come back in operation.

In general, Seryoga is with us a guru, somehow saved all the courts, but for now it is impossible to talk about it. He also had a long trip to the fire alarm system of a 22-story building, it was necessary to integrate it with all the systems. For example, with evacuation elevators, warning and about a dozen more subsystems. And even among themselves, these subsystems make friends. He then had three weeks of separation from the family and the dining room. He says the second was particularly hard.

Communication when how. Somewhere normal, some not. For example, in Magadan, Wi-Fi was through the hotel’s cash desk — a password was printed on the tape for 1 hour or two. And this picture in all hotels of the city - that is, in both. There is also a local 3G, but it still jumps over the satellite, so everywhere we try via Wi-Fi. Although in the same Surgut it was easier to distribute from a mobile - faster and more productive than from the local network. Our sims are corporate, paid by the company. But not everywhere our operators catch, so sometimes we take our own, backup ones. For example, in Noyabrsk there was no network at the hotel, and the corporate MTS was bad. Locals say, Mega works well - I bought a sim card on the spot and a great surfing trip for 200 rubles.

If we have a laptop with us, then in general we are able to connect to local closed networks with technical traffic, if it’s bad with the connection.

Cleaning other people's jambs

Sergey appeared on our doorstep in 2006. He graduated from the Voronezh State Department of Radio Engineering. He worked in the regional clinical hospital, served CT and MRI. Then he was engaged in security in Voronezh. It was necessary for family reasons to move to Moscow, I began to look at which companies in the market are in his profile. Suddenly I learned that a lot of people are required for intelligent buildings (this is just an engineer at the junction with IT). Came to us, passed the interview, got into our team in 2007.

Seryoga often gets objects where it is necessary to “clean out” after another team. Usually, how - a contractor works, promises mountains of gold and commissioning just in time. Maybe we have this specificity, but “just in time”, it seems to me, is a professional myth. A couple of weeks before delivery (or in general in 2-3 days), the customer realizes that if there is no physical infrastructure, then they probably will not have time to pass. And they call us. Here all the words are remembered, and about the "astronauts", and about the "special forces", and that only good things you do not know about yourself. In 80% of cases we decide, but it also happens that the customer is surprised:
- Three days have passed! Why not ready?
And we are sitting and thinking - really, why didn't the three of us build the missing buildings?

Or another example, we finished it there after the official launch. There were somehow works on commissioning of the systems - there 10 people after 9 months did not give birth to anything on time. They handed over the object, because when the fire inspector came in, the customer proved that everything was even, showing army ingenuity. They put the fighter on the roof, he watched through binoculars what the inspector was doing, and on the radio reported to the “Soviet automatics” - to the other fighters in the panel room what to include. As a result, they turned on one of the pumps and forgot to turn it off. Water filled the roof, and then began to spill onto the street. There is a center, cafes, everything - the people began to panic and google about the rainy season.

Far East

Pavel came to us in 2007 after the army. Then we just recruited the interns, he settled. He says, I remember, they gave me a manual of some kind, it seems, the words are familiar, my topic is professional, and no one even showed me this equipment for 5 years of the university. After the trial period, the first "combat" business trip - we went with him to Rostov to launch a store. At the airport, two flights were combined, and on the spot it turned out that our luggage was forgotten to be loaded (everyone who flew by the second plane). And we had a tool there. The customer says - most importantly, they arrived, we will give everything. And gave. The luggage was taken away only in the evening, but it was no longer very useful - they had fulfilled what they were.

The most memorable business trips are Magadan and Khabarovsk. On Friday, flew to the office, on Monday flew to Khabarovsk. The difference is 8 and 7 hours. In flights - 9 hours there. Then another 8 hours. The work is the same - air conditioning, UPS, diesel. It is necessary prevention for the data center and server.

When you fly from the Far East, the day increases by 6-8 hours. At 5 in the morning I got up, at 5 in the morning I flew to Moscow. But this is still good, usually this flight - at 3 o'clock in the afternoon took off, in Moscow at 3 in the afternoon too. If you are not lucky - an accident, and you have to go to the office. Or vice versa, when you fly out of Moscow there in the evening, then in the morning on the spot. I didn’t sleep well, but already morning, I have to work again

Anniversary Pasha celebrated in Novokuznetsk at the facility. There it was necessary to change the equipment, the customer said that he was finally delivered (hurray!), But it turned out that he was guided by the report of the transport company that it “will be tomorrow”. I waited another week. All night he was woken up by calls with congratulations.

By the way, about frosts. I somehow got the most frosty day in Kemerovo, when it was either -42, or -43. I knew it would be cold, but I had no idea what it was really FROZEN. After this incident, we agreed with the resource allocation service; we bought new Bavaria jackets, warm ones. It's easier with them, before that it was still necessary to “hook-up” to hook.

Took - do to the result

Pasha (the other) entered the team firmly after he proved that “There are no impossible tasks”, and he will decide the question before the victory. This character trait is very important to the engineer.

For example, I had the following on the road - literally last month, the opening of a new large store in Surgut. It is necessary to climb 10 meters to put some communication. And no stairs. They took and rented a picnic - a lifting mechanism. At this time, the temperature outside has already reached -20. They come, begin to heat diesel, they say - "climb." I:
- Guys, you hydraulics, then chase, frozen, I guess.
- No, everything is fine, let's work.
- You run, lift and lower, we'll see.
Similarly, it comes to the middle - the hose tears off. Everything, tragedy, repairman wait. They had to repair the picanick on the spot and work in the shopping center. Pasha's stories are not so bright, but I think the idea is clear.

In general, Pasha worked as an assembly and welding technologist at one of the defense enterprises. He dealt with large welding machines and not only welding machines and units. When I first got there, the worldview turned upside down as the workplace might look. After a huge shop in this enterprise of such a half-dark where 3 pensioners worked (not the best years were, the end of the 90s), a normal engineering team with a modern tool was like a toy store for him.

Before us, I was looking for work with difficulty. “We need young cadres, but go somewhere to gain experience.”

Pasha also noted that he went to an object where there was no electricity, and all the BP were dead. We had to carry the batteries, and he just took off the plaster. Worked on a par with local, all repaired.

And on the first trip, he went to Baikal, to Aldan, but this is better to tell in bikes.

That's what I told about:

Our team: Seryoga Zolotarev, Pavel Alexandrov, Pavel Stepanov, I (Roman Karpov), Anton Timoshenko and our leader Alexei Vorobyov. He left earlier, now almost stopped. We are still young in comparison with experience.

About Anton and Alexey did not tell, they are better themselves. They will write about our stories.