Romance of long-distance engineering trips (and our rat survived)

"- Come on, Che, is it really wrong? Conveniently, the cable is stranded, soft. You can't wrap it in a single-core one ..."

The customer - a big company - announces a tender for UPS and air conditioning for the infrastructure of its data centers throughout the country. Objects slightly less than 30 pieces, and including the city, where there is no railway. Basically. In August, the competition starts, in November, just finishing all documentation. Accordingly, our engineers arrive at the very moment when it is outside from -30 to -40 outside air. And under the terms of the contract you need to wash the outdoor unit air conditioner.

Who does not know, I inform you - it is done with vodka . It is desirable - burned, it is frosting worse. The procedure is this: buy a box or two in the nearest store, galvanized bucket. At this moment, passers-by are already beginning to react extremely painfully, watching the discharge of vodka into the bucket. Vodka, by the way, in many remote cities is still an alternative currency stronger than the ruble. Then, on the faces of those around, genuine sadness and all the world's pain appear when the air conditioner unit begins to be flushed.
')
We always send at least one engineer with such experience to such trips, because there are a lot of pitfalls. Starting from the fact that for blasphemy can be trite, and ending with the fact that a box of vodka and then “at home” must somehow be carried out in accounting. And there will be questions. Another danger is that when spraying onto the external unit of the air conditioner, you can breathe in alcohol very quickly. As experienced jokes - in the instrumental set it is necessary to have a compartment for cucumbers to remove the effects of this type of inhalation. The next problem - everything needs to be done quickly, because at -35 and below, taking into account the temperature of the condenser about 60 degrees in positive territory - the smell of vodka spreads rapidly around the area in the direction of the wind. Or omnidirectionally, if there is no wind. Locals really start to behave very strangely and look for a source of concern.

Come in, I will share other stories.

Business trips with us every week. Frequent, but short. For 10 years in the department, we have already visited the entire country. And also in Kazakhstan, in Ukraine, on studies in Holland, China, Germany.

Away from the usual routes of civilization, there are a number of places that can only be reached via a winter road — a road that rolls over snow and ice, over lakes and hills. A winter road in Siberia, for example, is generally the only way to get to a number of settlements. As soon as the winter road melts, the city is literally cut off from civilization to the next cold. In the spring or autumn, in principle, it is impossible to get to the engineering objects, except that a helicopter in extreme cases. The distances are quite large - it is necessary for 700 kilometers through the mountains, for example.

On winter departures, the equipment is taken rather strange for “first men”. Three spare wheels, guns, axes, spare parts. If the car breaks down in the taiga, first you can never meet anyone for a couple of days. And then the teddy bear will come to the pleasant smell of gasoline and matyuki to ask who is so tasty here. The last time the driver at all was waving a brand-new "Saiga" with the words "shoot back from the locals," he says, the places are deserted, they indulge. The nature there is very beautiful - but after an hour it already bothers to look out the window, about the same thing.


My colleague catches himself a dinner sysadmin dinner

You need to drive quite regularly, because everything that cannot be done remotely (installation and installation) is better done by yourself than explained to the local ones. As a rule, they are good at minor repairs, but do not have the necessary qualifications, so we go to remote repairs only in extreme cases. So, since a qualified engineer is a rare person, it doesn't make sense to send more than one. As a result, on such trips the driver becomes the closest person. And even at some point, the connection does not take, so before the trip you need to stock up with anecdotes and service manuals. The last time we were moving from Neryungri to Warm Key (this is 500 kilometers to Yakutsk and another 500 towards Magadan), a river without bridges, we need to wait for a winter road. The driver decided that he would slip when there was no official winter road. The driver was fined 300 thousand for transporting people.

The cold, except for the features of washing the outdoor units of the codingists, brings quite a few surprises. For example, in Khabarovsk we were told the following story: on the spot we decided to paint and insulate the expansion tank of the hydraulic system. As expected, planned for 2 hours on a simple. But did not take into account the transition period. At 11 am in May it was 10 degrees. After 2 hours it was already 30 degrees. Thermal expansion of the liquid has begun. Capacity in two and a half tons just ripped at the seams. Brewed, started running - whistling. As a result, we learned all this background when we mounted a new expansion tank.

A lot of shoals opened on the ground on trips to new installations. As a rule, you come, you look at the project - there is one, and in fact - a little more. That's literally not far away, in Moscow already the Serbian subcontractor of the customer handed over the engineer. We approach the outlet, check - there is no voltage. The customer opens - there is a wire, everything is neatly mounted. Pulls - he falls. There is a 30 centimeter tail, and this tail does not go anywhere else. Apparently, Tesla's secret to the contractor was not given. Or, on another facility, subcontractors connected our fluid pressure sensor. It has two liquid inlets and an electrical inlet to take readings. The contractor’s engineer did not understand. One liquid line is connected correctly, the second is connected to the electrical input (a hydraulic tube is inserted), and then the phase is connected to the second liquid input.

Somehow it was necessary to connect a small UPS. The cable should be according to the standard about 16 square millimeters. And there goes the cable line 3x2.5 mm. Then there is an automatic switch. From there towards the consumer grows up to 3x10 mm. We:
- And this ... you like everything here?
- Do the drawing. So approved.

Last year I went to Khanty-Mansiysk. Asked to see the equipment, something too quickly failed compressors. We leave on the platform, around the uncle with ties, obviously not the last chiefs. Go, look. The dialogue turned out like this:
- look? Here the compressor often burn. We change them. Everything else works, sort of. And so since 2007. Mystic.
I look closely and gently report:
- Gentlemen, and you are not alarmed that the inscription upside down?
They:
- How so? Can not be!
Look at each other
- Vasya you look ... In fact!

All external air conditioning units are upside down. Compressor suffers under undue load.

Khabarovsk - there someone implemented a scheme for switching on circulating pumps for 15 kW directly from a switch without starting elements. 4 pumps were burned during the month. “I turn it on, knock it out right away, I turn it on again.”

Or here comes the commissioning of the UPS , and the installation was done by a local contracting organization. I checked. I look - cable lines are not made according to the standard: the ground must be yellow-green, but it is different. Phase conductor in place of zero. I opened the battery cabinets - there were a total of 124 batteries for 4 battery lines. An ordinary normal person would divide by 4 - it would turn out to be 31 per cabinet. But no. In one 24, then 28, then 36 - then the rest. I try to understand the logic so far. At the same facility there was a small item with a UPS on another floor. The installer says: "The extra jumpers remained." And the vendor does not put anything extra in the kit, this is important. We decided to peresherstit. As a result, it turned out that if we had supplied power during its assembly, we would have remained without a UPS.


Quite a frequent sight at the facility

I'll tell you about the usual road adventures. In the south also has its own specifics. We made the infrastructure for the customer in Dagestan. We're going back - at the crossroads there are uncles in camouflage with army machine guns. In the news afterwards - only our engineer left the facility, there was a skirmish right there, just 15 meters away.

In another hot city, a heavily-removed transformer was left on the road until the morning; in the morning it was necessary to remove and rewire it. In the morning we come - it is not. But the road is new, the asphalt is laid where it lay. And it was impossible to carry without special equipment. Began to ask. It turned out the following: the official was supposed to drive along this road according to the plan. Therefore, around one o'clock the workers came and dug a hole under the transformer, "level" it on the road and laid asphalt on top.

In St. Petersburg, there was a story worthy of the cultural capital. The customer was located on the territory of the power plant built in 1901. We had a briefing for 2 days (TWO DAYS) just to be allowed to work. The lady was very fond of the power station and, therefore, told us first the history of the object, then the area, then all of St. Petersburg. Already got to Peter. After the admission, we exhaled and went, but it turned out to be not everything - it was possible to walk only in the inner zone. But on the street where the danger zone - another 3 hours of safety equipment interspersed with history.

In the same place in St. Petersburg, we had to get a 5x95 cable at the site of about 6 meters . And dragged the hundred-meter tonnage coil. They say: “It can be useful. Not useful - well, we'll take the trim. "

Once repaired DGU , which simply did not get up to service for years. The customer signed the delivery, and the service has been coordinated for more than two years. Service installation no one thought. When they got up, we found rotating elements erased to meat.

We saw a cooling data center implemented backwards. Cold air entered the exits from the servers. It was supposed to “reverse” the installation of the racks, but cooling was done by one team, and the iron was put by another. We comment on:
- You will feel bad here.
- You do not understand, the manufacturer himself designed.
After 3 months, of course, backfired.

In Serpukhov, I come to the site. On half of the future object there is no roof, inside the puddle. We approach the place of installation of the server - they are preparing to pour the floor, there is cement. I say: “In such conditions, iron will not work for a long time,” and me: “Mount, we have a schedule. We need to run the UPS on time. You have a contract, do it! ” In the end, they did as they demanded, and then several times they rode unplannedly.

A colleague talks about the shortest launch:
- Do not touch, now burn!
- Run, we have a schedule!

Or it often happens like this: “Pass the object, just don’t turn it on”: for example, on a large number of objects with gas fire extinguishing the initiating devices are not connected to gas. Because suddenly there will be a drawdown. At least one data center burned down for such a reason I know.

But let's about the good. In general, there are prudent customer representatives who know exactly why a raised floor is needed. They have there in the server downstairs lies vodka, cooled. One of the manufacturers of ventilation saw a small server with properly equipped hot output. They decided not only to warm the environment, but also to correctly use heat for their own good. They put a small brewing unit, rejoiced at their home.

Once the customer’s engineers had to urgently solve the task that the manager had set at 8 am on 2 January. “I don’t see whether it’s warm at the facilities or not. Put the weather stations and monitor them in Moscow. " After 4 hours, we saw a magical picture - young funny guys were hanging thermometers near the security cameras. Turn on the video stream - the temperature can be seen.

A horizontal bar hangs in our office. The fact is that we need to not only run away from the bear, but sometimes also carry a lot of things. Therefore, we play "ladder" all the time. Of the cases of dragging, the most remembered one was the DGU on the roof of one of the distant data centers; 3 cars of 200 liters each had to be pulled in with buckets. Initially they wanted to make a hydroplatform there, and then they simply hacked the second part of the financing and decided that they were better off. We once participated in this circus, and the service team has fun there regularly. Free fitness.

Once the customer (big order) personally went to watch the switch on the UPS / DGU. Opens, there is a relay.
- Ahhh, arc!
And jumps off. His:
- Calm, normal working situation.

The big boss thought and the tour was interrupted. Asked further in the papers.

In Voronezh, repairing shields and UPS. I say: turn off the fire sensors, dust may rise, the sensor will work. They have not disabled anything. I take a blower, blow into the fan. A huge table of dust rises exactly up - and there is a sensor. It fires across the entire facility — I'm running to the security to say it's dusty. "I know nothing, complete evacuation." The chief engineer came running to mate: "And I told you so." But in the end, the customer praised everyone. He says that while you set up a circus here, the colonel of the Ministry for Emergency Situations went to the trade section. I saw that everything works like a clock - it will not come in this half year.

Once our foreign colleagues drilled a wall according to our project into the room for fuel tanks. And it must be said that the diagram had a characteristic “cloud” of callouts with an increase in scale, so that it could be seen which hole was needed. So these comrades exactly along the contour of the cloud and cut it into a natural size.

And finally, about the rat. For colleagues, it exploded on the battery line from surging feelings and tension, and our beauty here is:



Ours survived.