
Working in a large IT company on duty in the data center department imposes its imprints on lifestyle. In the night duty was born this story. I present to you the it-parallel with the Strugatsky brothers.
NIICHOSHI. Night duty- Where are you wearing? - the voice of the boss was not childishly stern. - You throw it to me, why not yet in place?
The call caught me at the entrance to our NIICHOSI (Scientific Research Institute of Private Objects of Information Systems) where the data center was located.
- Evgeny Palych, now I will, I’m already getting up! - in the handset of the mobile phone the hooters sounded reproachfully. Hmmm, well, well, I'm not late yet. Today, my first combat duty alone, I could not be late for him. I kicked open the door and flew into the office. Yevgeny Pavlovich, as always, in his checkered shirt and faded jeans, sat at the computer, continuously watching monitoring, and only for a second looked at me.
- Take a shift. While everything is normal, there are no problems. Order you know, but once again I remind you. Tonight no updates. At twenty-three zero-zero you look through all the cameras, whether there are any overstaffed employees. If someone is sitting, throw a message to them so that for five minutes they are saved, while he himself does shutdown after three. Everyone knows, let them not complain. Then you go around all the servers. Access vpn for all cover. Here is a list of those who can connect to our network this night.
He threw me a list in Skype. My phone stumbled and a list of twenty users appeared in front of me. The general director Boris Vorotin was in the first place, Evgeny Pavlovich Linusov himself was second, Konstantin Dzhobov, the senior system administrator, was listed third, and a list of seventeen user logins followed. Of course, I have been working here not so long ago, but I saw these logins for the first time.
- EEE ... and who is on the list from the fourth to the end?
- This is a list of those who are allowed posthumously access. Heh, just kidding of course, these are old accesses, there passwords have changed a hundred times, still no one will enter and in general this is the old list. Leave the first three lines. Okay, let's continue. - he glanced at his watch. - After three minutes, you start to be on duty and watch no one at the computer, even if you make your way. The guard passes around the perimeter every hour, he knows, he will not knock on you. Cleaners chase - yesterday already and so one somehow tore two patch cords from the rack, although the wires hang at the level of individuals. And I, mean, went. Holiday greetings!
')
Yes, my first solo duty was on New Year's Eve.
Pavlovich got dressed and left the office, I traced through the cells that he had definitely dumped. Brewed some coffee stronger. The night promised to be interesting and calm. I did not have time to take a sip, and the company of system administrators burst into the room.
Three guys from the next room. They seem to have something like a dress code - all with a beard. One in a jacket on a naked body. On the neck hangs a lath of a captured operative on one hundred twenty eight meters. Hands hung with baubles from vitukha. In his hands he held a tambourine, and, judging by his appearance, he had just performed the rite of resurrection of the server. This is Kostyan, he has been working here the longest. Someone says that even longer than the general, someone in general, that the data center was formed around him.
The other two are dressed more decently - today they came even in formal sweaters. Since the New Year is with deer. This Potap and Cyril. Specialists on niksam and tsiska (interesting or tsyskam), ardent supporters of automating all possible actions on all devices.
- Oh, Sanya hello. You chtol today for the beholder? Ah, something Palych atrocities. Can guys, substitute it for pervosti something? What young people will sit here? Let him go, celebrates. - Potap looked at me with regret.
- And the truth is, Alexander go home, we will stop, and the fact that you have a holiday is not a holiday, and now I will write down the script so that Palych will come to you with a report. - Cyril readily rushed to the first computer that he came across and had already begun to shave off to some kind of piece of iron.
“Come on guys, I still need to finish the nginx config, and edit and start it on the thirtieth, so that this year we can complete the plan.”
I really had things planned for that night, although of course there was not enough time on duty at all.
- Well, if necessary, so come on. Good luck, I’ll go to the server room for, hmm ... I’ll run for the thermal paste and go. - this is Kostya already addressed to his.
Yes, of course, for pasta. Everyone knows that they are cooled under the server kondey in the server pivtse.
- You if that, call. Or write. Although on the mail, even in Vatsap, at least in Viber, even in Jabber, at least in ICQ, at least where - you know that we are always in touch! - said Kostyan with a smile and pointedly knocked out his phone.
The whole trio of fun fell out of the office. There was silence, it was only audible how the employees' systemists noisy rustle and the office lighting lamps whistle. I turned on the browser, unsubscribed in our chat: "I took the watch." Just started the putty in order to check the servers, Anton Shtenter, nicknamed Brain, looked into the office. He was my colleague in the department, and by position - a leading specialist. He went a long way to this position, starting with a simple technician. So he has plenty of experience! For myself, I called him a walking Wikipedia - that he did not ask from IT knows everything, he came across everything in his work. Today he was dressed in a strict gray suit, it was immediately clear that after working straight for a holiday.
- Hi, Sash! Are you on duty today? Filled dock transfer duty? How is your scripting? - Anton piled on me with questions.
- Hello. Yes - on duty. I did not have time to fill in yet, I came only. And what about the scripts? - I asked. I began to master python recently, and any questions on the subject of writing code were perceived hard. I did not want to seem inept in this area.
- You come on, train. In our business, oh, this is useful. Well ka, let's write me a script cleansing all the schemes based on geo-systems with the condition that the system is three-level and consists of five modules. - Anton rolled the nearest chair and sat on it backwards, riding like a horse. - Just come on for a while, on the test, but I would not want to spend the New Year's Eve at the elevation of the sysadmins from the table and the bases from the backups.
Then I tried to slap my work for half an hour. I didn’t want the script to start as normal. That invalid syntax, then the expected an indented block will come out, otherwise the computer will freeze without explanation. We must pay tribute to Anton - he held on well. Once he just uttered in an awesome whisper: “Well, where are you shoving for that? While this is what is needed ... ”In general, I fought with this script, and Anton did not agree to accept the work until at least half of the task was completed, and forced me, to rewrite it again and again. Finally, I learned something similar to the code that can be executed and not carry all the data without a trace everywhere, but only what Anton asked. That's the same, - he said, - practice, you do not have enough Sasha, practice. Well, okay, I ran, I still have time to take the subway. And you see, while with such skill on combat servers, the scripts are not cool. Holiday greetings!
A phone rang in his pocket, and he put on a black coat and went out.
Having exhaled, I quickly did rm –rf on the daddy with my created script, so that someone could not find it. He glanced briefly at the monitoring — a couple of red triggers and a dozen yellow ones were burning there — no problem, it's all within the bounds of normal. You can still see software developers frolicking, perhaps already taking a bit in the evening. Nothing soon according to the plan we had them ... All systems worked properly, it was time to read Habr. But first, I filled out a document of the transfer of duty, nothing complicated, just need to. I quickly ran through the server consoles, checking the load. Of course there is monitoring, but according to the rules, for some reason it was necessary ... That's it, now you can read it.
I managed to dial only “hab” as Boris Mikhailovich Vorotin himself entered the office almost noiselessly. As always, our general was dressed in a perfectly clean strict black suit. In his hands he twisted several keys, as a trinket served two, punched in the middle on through the ring, a token with old EDS. Behind him barely squeezed through the door, the head of the development department, Robert Mikhailovich Stollov. Faded trousers and a slightly soiled shirt with the weight converged on his very small belly.
“Oooh, Aaleksandr Paavlovich, good evening!” - greeted me, slightly sipping vowels, Robert Mikhailovich. - How is your insanely kraivivy Aangor rabbit (xs, from which he took that someone I have. But everyone is already accustomed to his unexpected questions asked not on the case)? To your note, I ask you not to forget and follow my script on the main combat database for the deviation of information. It should end in the area of two nights. In the process of execution, you must carefully follow the free space on the partition. And the whole swap can also be activated, if you don’t have time to beat the “leis processes”, the script eats all the memory. Be aware, this is not just a script, this is Perl art production! And if you need more space, do not be greedy, add more.
I was discouraged.
- But, Robert Mikhailovich! I was forbidden to run anything tonight! According to the plan, only the monitoring of work and no updates.
- Well, my young Padawan. This script runs already legendary twenty five days! Can it really be stopped at the Saami finish, with the trembling hand of the young attendant? This task is from the very ... - Robert Mikhailovich raised his index finger, pointing, somewhere in the direction of the ceiling. - Execution of it will lead to the fact that it will be possible to completely refuse vacuum on the bases of our data center!
Gendir, who was standing nearby, looked at Stollov in disapproval. Naturally, he did not give any assignment to Robert, moreover, from somewhere “from above”, this thought occurred to the PR head. Though he was a weird person, he was not even going to be dismissed because of his experiments, he knew his business.
- Follow the script. Categorically strictly follow the monitoring. In case of an accident, act according to plan. - As always, I personally personally put Gendir himself briefly and clearly set the tasks and went out just as silently.
- Taak is a young man. As soon as it ends, it means the script. Call me right away. I'll log it out. Since it is necessary in the report to bring all the splendor of my great idea.
Robert Mikhailovich turned around heavily and spanked to the exit. The door squeaked in response to his pass, and I was left alone again.
The office smelled of Gendir's light fragrance, mingling with the heavy smell of fumes from Robert Mikhailovich. Yeah, everyone started to celebrate even in the morning apparently. Finally, I was not supposed to disturb anyone. Now it’s all gone, I looked at the cameras. Nobody had to chop off. While everything is calm, I took up my favorite thing. Reading articles on Habré. A lot of things today published. In the middle of reading one of these, the work phone rang. It's almost New Year's already strange, first of all, secondly, we do not provide technical support services, and thirdly, it’s too lazy to do something. But according to the rules, I have to pick up the phone, no matter what happens.
- NIICHOSHI, the administrator on duty Alexander, I listen - I introduced myself by the rules.
- Aaleksandr, my friend. I forgot to stop you. - I recognized the voice of Robert Mikhailovich. - Be attentive. The script triggers the network, follow the load. Otherwise ...
There were short beeps in the phone. Head of the OR went down the subway. He did not particularly think about others, so it was in his spirit to end the conversation in this way. Oh well, the network is so network. Well, he will not change the ip-shnik anywhere, probably just the rules in the firewall touch. I continued reading. The phone persistently strummed again.
- NIICHOSHI, the administrator on duty Alexander I listen.
At that end is silence, although I felt a slight sniff. Such a stupor sometimes happens when someone makes a mistake by the number and gets to us.
- I listen - I lost patience, I was waiting for a great article ...
- Hello, this is your client from Norilsk, happy New Year, by the way. I am unable to connect to the web interface of your cloud to make calculations for our center. - the feeling was as if he was reading a piece of paper, perhaps he was left with instructions on how and what to do. - And in your technical support do not respond. And I urgently need!
Fuck you! Urgent to him! In the New Year! Fuck ?!
- One minute I will check. - I got to the camera located in the technical support department. So it is, the only person on duty was sleeping under his desk, with his face buried in a sistemnik. On the table was an unfinished Jack Daniel's bottle. Poor thing, for the last five hours not a single call, got tired.
- Tell us what exactly you are doing and how you are not working.
- When I left the house, I was twenty minutes late for the bus, so I had to take the subway. Then, when I got to our office, the guard did not want to give me ...
- Stop, stop. Not so detailed. Let's go straight to the beginning of the problem.
- So I sat at the table. Pressed a button on the processor. He zazhuzhal, and all. Black screen and I can not connect to your site.
My hands dropped from the keyboard, which were already typing in the tail / var / log / ngi console ... I looked again with hatred through the window with the broadcast of the sleeping support. It was obvious that he, stsuko, smacked his lips, and his hand twitched nervously in the dangerous vicinity of the reset of his computer.
- Press the button on the monitor on the side and it will work.
There was an intensified sniff in the tube and, it seems, even a stirring of the convolutions (s) of the interlocutor's brain.
- What is a monitor?
- This is the box you are looking at.
- Oh, thank you very much. It worked. Once again - with the Coming.
He hung up. I sat in shock for a moment with the tube pressed to my ear. Well, customers with us.
Suddenly, the monitoring system turned bright red. About a thousand red triggers roared that everything was bad. My short hair on my head began to rise on end. But I smoothed them with my hand and calmed myself. Most likely, this is just monitoring buggy, something like this happened. It would have been enough to restart one service, but I decided just in case to completely reboot the server. Everything calmed down. Since no more problems were observed. I began to walk around the Internet in search of interesting news. In the corridor was a guard. His gray head through the muddy glass of the wall was like a ghost. Therefore, among his own, we called him Casper. By the way, he braked, just like the eponymous antivi ... The ringing of the phone tore me from my thoughts. What kind of a night is this? He never called me more than twice a night. And if he called, the second person on duty had already taken the phone.
- Hello again. - the interlocutor ahead of me - This is me again, from Norilsk. Sorry to bother you again, but your tech support is still not responding, and I have a problem.
Glancing at the camera from the TP department, I noticed that the guy had already crawled a little under the next table, put someone's replaceable shoes under his head and in a dream bit the wire from the surge protector.
- What happened again?
- You see, when I earned your site, I went to make coffee. The coffee machine was turned off, but according to your advice, I pressed the button on the side and it all worked. I put two spoons of sugar and put the cup on the table and when ...
- Stop, let's go straight to the problem!
- So, and when I drank some coffee, I began to enter the data of our personal account. And everything goes beyond nothing. I have already tried, as you advised, to press a button on this ... on the monitor. But it just goes out and has to be turned on again, but the problem does not disappear!
- What are you entering and where?
- Where the lodin is written, I enter the name of our company. Where there is password, there is our password.
- Well, firstly, the "login", and secondly, you need to enter in Latin. Thirdly, you have not included caps? Or maybe you’re just wrong ... You know, let me remotely connect to your computer and do everything myself?
- Is that allowed?
- If you do not mind, then of course you can.
- Well, let's.
Fortunately, on the computers of our regular customers (and this was permanent, as I looked at the records in the admin panel), a remote connection system developed by our programmers was installed. I quickly found his only computer on the HUHA system (help user the hands of assholes). And connected.
- Oooh, my mouse moves itself. I seem to have caught the virus on your site!
- Calm down, I connected to your computer and do everything.
- Cool, and it happens?
- Well, I did it like that.
Quickly entering the required data in the fields, I clicked "enter". And now the main page of their personal account on our “cloud” has opened. Everything seems to be breaking further and there is no place to mow ...
- I did everything, then you yourself.
- Thank you for what I would do without you. - on the other side hung up.
“You wouldn’t live to that age ...” I said, turning to the beeps in the receiver.
This, of course, is not our job, and I gladly thought that I had begun my career here, and not a technical support employee. By the way: I looked at the broadcast from the TP cabinet. The window was black and in the middle there was an inscription: "The signal is lost." It was strange. The video surveillance system has not failed for more than a year. I got up and went to the TP office on the fifth floor. Still rising on our transparent elevator, I noticed that their whole office was in complete darkness. Do not even light, as always, not turned off monitors. I began to guess what was the matter. , 112 , , . . , . . , , . (???) . - : «…, – …» . -, . . . , . . . , , . – , .
, . !
. , ip- , !
— , , .
— , . - , , - ! . Such situation.
I almost started the calculations, but then my wife called and when I took ...- Let's go ! Immediately! To the point! Problems! - I almost lost my temper.- So, when I picked up the phone, I accidentally hit a wire, and everything went out. I do not know what to do! Here is some wire pulled out, I hold it in my hands.- Is this wire black? Does this wire have two pins, such as a wire from a microwave?- Yes Yes. That's right.
“Then you need two clips.” Do you have a number of ordinary paper clips?- Now I'll see ... Yes, here they are!- Now listen carefully. Take two clips. Unbend them until smooth. Then take one clip in your left hand, the other in your right. And gently insert into the holes on the surge protector - this is where the sockets.- Isn't it dangerous?- Would I advise you if this were dangerous?- Right. Oh well.
Straightened. I take one, another. Inserting ...In the tube I heard a light pop . I hung up. I am not an evil person. I wonder if I can call an ambulance to Norilsk from here?For ten minutes I enjoyed the silence in the office. Then I went to the cooler for some water. Climbed on the server telephony, deleted his last conversation. Why? . , . . … ? ? . , . , ssh . , . . . .
— . ? ? ? , ! ! !
- I dont know. . ! – — 02:00. . – , , . , .
— ! , . , . . .
. , . – . . .
— , !
.
— … … — .
, , . . , . , . , , !
. , . :
— , , , , !
. . :
— – . ! , , , …
, . .
. – . . , , (, !!! ) .
— — ? !
— . , , : « , »!
— . - . , . , . !
— . .
.
— ! ! , .
, «», , .
— , , ? – . . — : , , ! ! , . . - . …
- , … . . , . . . , , - . , .
, .
— ! ! , ! — , , . – , . , . , . . .
. . .
. : «, … …». . . , .
- In short, look. I have almost nothing to do here either. You start everything on the sly. Raise the base. Checking services. I checked - access from outside appeared. Call me if I can. I will not turn off the phone. - he smiled, stuffed a tambourine in his belt and left.I was left all alone with the servers, half of which were not yet running. Punching through the power buttons, I stumbled upon Kostya’s phone. Damn, I wonder if he does this on purpose? After everything was started, I moved back to my workplace to run everything programmatically. He looked at his watch - half past five. I have only three and a half hours to lift everything up and arrange an accident. I wanted to sleep after such nerves., , . postgresql . - . - . , . , - . . . , .
- Hello. Is this duty Alexander? - I recognized the already boring voice from Norilsk. - You know, I got something bad here and I was cut off. When I woke up, it turned out that everything was working. Have you deleted my computer? Thank you very much for your help. You know, sometimes in your technical support such rude people answer! But you will always help. I started the calculations. A little everything slows down. Maybe you will log in remotely and see why my game is slowly starting on the site. I have already launched it in ten tabs, I think at least in one it will start faster. And yet, you know, it happens, you open classmates and ...I hung up. I turned off the phone. I put the phone in the bedside table. I locked the bedside table. I put the key in my pocket. It's just not his day ...Two more hours of my torment passed with the raising of bases. Registration of the accident report was delayed for a long time, it was necessary to provide all the details and time, what, who did it and when. Scared to sleep. Coffee no longer helped. By the morning began to gather message queues on the services of distribution. Urgently had to switch to this task. Suddenly, the provider disconnected the main Internet channel. Well, yes, of course - today is the first of January!I had to quickly switch everything to backup.I squeaked the door to the office. Max came - on duty, who will replace me. Yes, it's almost nine in the morning.- How did the night go? I heard there were small kosyachki? - Max asked, yawning. - It seems there were no updates ...— . , . , .
— , . , . .
— , ! .
, , , , :
« ».