As I drove the robot, I almost turned gray and filled the server room with blood
This is the story of one of the most memorable cases in my engineering practice. For obvious reasons, I changed the names, places and some recognizable details so that it was impossible to pinpoint the customer and other participants in the story.
This is how a tape storage looks (ours was smaller) and a library robot (ours is the same). Chinese not included.
I remember, there were the last days of November. Already thinking about the end of the working day, I was planning my evening, when suddenly I was informed that in the glorious Siberian city N our customer had a tape library broken. The spare part was immediately sent by the transport company, but for 3 days already, as it was still on the way. The transport company was vaguely explained and hmakala in the phone, and the customer was not in a joke nervous. Forecasts were uncertain, so it was decided to carry another spare part on their own by plane. A warehouse employee handed me a size box weighing ten kilograms, pasted over with bar codes and stickers, and happily slammed me on the shoulder with the words: "Just do not even think of taking my luggage - they will remember."
')
The box certainly disturbed me, but I was equally worried about a bag of dry milk, which at the last moment was handed to me by the project manager. “They have some troubles with milk now ... because of bad weather or something ... local guys asked them to bring 2 kg. Well you are not difficult? "- He said. From the expression of his face and the characteristic gesture of the palm, as if covering my mouth, it was clear how he now wanted me to be compliant or even dumb.
This story happened a few years ago, when I was an engineer, when I flew around the country a lot, and when I woke up in another hotel I sometimes couldn’t figure out which city I was in right now. 4-5 trips per month left their mark on the way of life. And the embodiment of this structure was my backpack, which was always ready for the next trip.
I got out of a taxi, and suddenly a police officer stopped me at the terminal building. “Your documents!” He spat out, and for some reason, a feeling of deepest satisfaction spread across his face. An evil dank wind blew, preventing me from sorting out my pockets and bags. I, as I could, took out my documents and handed them to the law enforcement officer. In theory, a young man who travels only with a laptop, a large box and a bag of white powder should not cause any suspicion, so I depicted a sincere bewilderment on my face. “What do you have in the box?” He asked indifferently. Apparently the fact that the package was nothing like dried milk, he had already identified with his professional sense, and therefore he was no longer interested in the package. But the box has not yet yielded to his insight. “This is a robot,” I said.
At that moment I had a feeling that his indifferent face was replaced, because it was lightened from the inside with genuine interest, as if a module with the function of human curiosity was additionally connected in the depths of his cap with a cockade. I understood that I was in a hurry with the answer and it was necessary to fix the situation somehow. "This is a robot ... for the library," I added even less confidently, continuing to monitor the connection of the remaining modules of malice, gloating, and greed. "Speak, the robot is for the library ... is it a librarian or something?"
I moved to the landing zone, remembering how I had just finished with a cockade. “It cost,” I thought, “and could have been late for the plane.”
“We remove luggage for tape, jackets, belts and shoes, lay out our phones ...” - a woman in uniform commanded powerfully and soullessly. Having mounted my belongings on the tape, I myself without sound and peep passed through the frame of the metal detector. It was clear that the box with the robot was haunting the airport security officer who drove her back and forth through an x-ray. “What is it?” She finally said when all the comparative images in her head had run out. I crushed a bag of milk in my hands, pondering my tactics so as not to repeat the cockade episode now. But suddenly her colleague, sitting next to her, glanced at the monitor and said: “This is a robot for the L200 library. For this week, the third time being taken, the season of breakdowns they went. Come on in, young man! ” There was no limit to my amazement, but at the same time I was terribly grateful to my providence, as well as to the technical literacy of fellow citizens working at airports.
Rusty "Volga" creaked rushing me from the airport to the hotel. Of the thirty people of dubious kind of taxi drivers who met the night flight in front of the local airport building, this one seemed to me the most restrained. In all his slouching and smoky appearance, there was a sense of integrity and even nobility. However, the appearance of a taxi driver is often deceptive, and he wondered all the way why I live in Moscow, but I don’t know how Alla Borisovna Pugacheva is doing. “You live with her in the same city, and you don't know?”, My taxi driver did not let up. He disliked me for ignorance, but at the same time did not want to lose face with the face of the capital's guest. To the door of the hotel we drove famously and with a squeal of brakes, completing a steep pirouette, which is not allowed for an auto-assembly with such a venerable age. The door of the car seemed to open itself, or just fell off. Hearing my question: “Will you give me a check?”, The taxi driver waved his hands and completely forgot his conversational skills.
I went inside the building, looking around at the tastelessly decorated walls and pillars, all with some sort of gray bloom, like dust. The building was an old building, possibly a former manor. At the reception, the girl idly painted her nails. Confident in my own way as a client, I resolutely took out my passport and slapped it on the counter. Madame reluctantly distracted herself from her nails, looked contemptuously at my passport and muttered: "We do not give in debt." “What ...?” - interjections and other signs of extreme indignation ran in my head. “I have a room booked here,” I tried to be polite. “This is a casino! The entrance to the hotel is on the other side of the courtyard. ” Who knew that both places have the same name? And then I managed to think that the taxi driver had been driving me all this time, apparently thinking that I had flown from Moscow to this unwelcoming winter region to scour the contents of the box behind the cards and the tape measure.
The hotel room met me with chic curtains to the floor and a musty smell of damp. I sat down on the edge of the bed, located in a niche of the wall, gently lowered the cherished box with the robot in front of me and greeted me unexpectedly with a spider in the corner. Works at the customer were scheduled for 9 am, the clock was 6 local. In my head, the IL-62 engines and other charms of the many hours flight were still buzzing. I mentally figured the prospects to sleep a couple of hours and wake up fresh, and made the right decision just to stay up. Physiologically, my case was headed for a late dinner, even a kefirchik for the night, so I straightened our corporate green tape on the box and went into the shower.
When by 9 am I arrived with a robot and powdered milk to the customer, at the entrance I was met by a representative of the IT department, who said that his name was supposedly Semen. Semyon talked about everything around, wiping his mouth with his hand every now and then. By his appearance and rustic manners, it was clear that in the department he meets the role more often than others, as the most illiterate specialist, but a very sociable person. Without a doubt, Semyon inspired confidence in people, and, as it turned out later, had a wonderful, almost indomitable appetite.
There was a fuss on the floor and it smelled of paint. Rapidly gesticulating, Semyon explained that this is a new building and now a second site is equipped here. And then my mobile phone rang. It was the project manager. “Have you got to the customer?” Listen, help them there. We are now setting up a cluster. Our Lesha has a week. Something does not work. Well you are not difficult?
Accompanied by Seeds, I entered the workshop. Everywhere there were boxes of different sizes labeled Dell, Cisco, EMC. Many of them had already been opened, around lay the wires, rails for mounting equipment in racks, bales of documentation and electricians. The electricians were not just lying, they were swarming with wires under the raised floor and were quietly quarreling. In the center of this creative chaos stood the senior specialist Nikolai. He held a workshop. Hiding from the meeting, our Alexey languished behind a pile of boxes, squinting at the monitor of his laptop and sipping breakfast on the sly. Semyon also wanted a loaf, but he had to participate in the meeting, so he just puffed.
From the first words of Nicholas, I realized that the launch time of the complex was running out, less than a day. With the advent of the second site, management has set the task of reorganizing one of the most important information systems. The database — the core of this information system — was supposed to be clustered. Now it was stored on two storage systems, between which synchronous data replication was carried out. And the cluster software tracked the state of the whole complex and could automatically respond to a failure, if something bad happened on the main site. The preparatory work was almost completed, but a lot of time was lost in the search and elimination of broken optical links between the sites that the contracting organization laid. And still had to finish the installation of servers, upgrade the operating system and microcodes, and test everything. The launch of the complex into commercial operation was scheduled for the night - the only time when it is possible to stop the responsible financial systems.
I don’t remember how this day went. I only remember that colleagues immediately declared to me that he would wait for the tape library in the light of the circumstances, and now my hands are more important in another matter.
The whole day we mounted servers, twisted nuts, laid cables, connected adapters. In the breaks, we even seemed to have lunch and went for a walk in the forest. Then I put the operating systems, rolled patches, conjured with microcodes. Sometimes microcodes conjured with me, and in order not to nap, I remembered how on a plane a flight attendant skillfully juggled a teapot, complaining about the turbulence, and distributed tea to wary boiling passengers. Night was falling, from the window it was clear that a large orange moon had risen above the forest.
“Hour X has come” - Nikolay said cheerfully, yawning, briefly conferring with someone on a local telephone. "It has come, it has come," they repeated to him from a pipe, which he had not yet had time to put in, and giggled. These were the “appliedists” who were also on duty that night and upon completion of our work had to make sure that everything worked.
On the clock, the hands showed midnight. We came to the crucial moment when we had to switch the replication of data to another storage system, wait for the synchronization to complete and raise the cluster at the second site. This process was commanded completely by Alexey, a well-known storage systems specialist in our circles. Some more were aware of his irrational habits ... He smacked and copied the volume identifiers into the script, referring to the list provided by the customer. Half an hour later, when this painstaking work was finished, Alex touched the enter key for fidelity, and making sure that it was real, pressed it. Everyone waited anxiously for something, and the general silence was broken only by the crackle of luminescent lamps. I tended to a raised floor and I had a dream in reality, but I do not remember what.
I woke up because the synchronization was over, but for some reason everyone was not happy. Cluster services did not rise. I was called from the country of dreams, so I looked in the logs. Through the fog in my mind it came to me that the problem was with the disk subsystem. The logical volume group was imported with an error and the data was inaccessible. The reasons were not yet clear, but the prospect of a rollback to the old configuration began to emerge; Plan b". At seven o'clock in the morning, the system had to be operational, so there was no time to think. Then I could not even imagine that the tragic events of that night were just beginning to unfold.
I meditated another five minutes and offered to roll back to the old system. Reluctantly, my colleagues agreed, dejectedly thinking that in the morning they would report to the authorities. We switched the replication direction and gave the command to import the disk group on the old server. I had already calmed down and dreamed of getting to the hotel bed with a sense of accomplishment. And suddenly I heard an exclamation: "And here it also does not start." And inside of us everything broke ...
The fog finally disappeared in my head. It is said that in an extreme situation, the human body is mobilized. I was madly thinking what to do. The storage systems were fine. After confirming this, my colleague Alexey gave up his hand, signing his impotence, and went to watch a movie on a laptop. I have already spoken about his irrational actions. But formally, there was nothing to complain about.
The cause of the problems loomed quite clearly. Previously, additional volumes were added to the disk group by the customer; however, they were forgotten to be added to the list of disks that are replicated. When the cluster started, the disk group was imported, but the metadata was updated only on parts of the disks, while the new ones remained in read-only mode. Now the disk group configuration was finally mismatched, and the data is not available with either the new storage system or the old one.
When it comes to financial data of a huge organization, their loss is tantamount to a catastrophe. I immediately remembered a similar story when, at the dawn of my engineering career, I ended up with a repair from a single customer - a serious structure that was legalized in the 90s. They discussed that their admin unintentionally deleted something. “And what happened to him?” - among other things, I asked. “Well, he is still sitting here in the basement, the third day already. We are waiting for the general, he will decide what to do with him, ”the skinhead officer with a heavy chain on a bull neck answered me quite seriously.
Have you ever felt your hair moving on your head, going to turn gray? Pretending that we lost the database, I didn’t even think about what would happen in the morning.
In situations such as ours, it is necessary to immediately call the manufacturer’s technical support. It is, at such an hour, of course, English. I roughly sketched out keywords for myself and dialed a number. Two beeps and the English baritone answered me. I told immediately what we are and what happened to us. Baritone patiently listened to my monologue and asked: "What is the first name?". I was taken aback, but, having collected my thoughts, answered. Then questions about last name, customer name, city, location, serial number followed and so on to infinity. Time ran inexorably. After finishing the survey, the baritone asked me to record the number of the request and told me to wait for the call of a specialist. The specialist did not call me, he wrote me an e-mail in 5 minutes. He wrote that he accepted my application to work, but his shift was over, so I need to call the dispatcher again and ask him to assign an application to another specialist. It is not clear why some people believe that they do not like to work in Russia. It seems to be a delusion.
And suddenly a miracle happened. Another specialist called me back, but he was from India. The second global technical support center was there and they themselves accepted my application. I was glad to him as my own father, but my joy did not last long: we did not understand each other. It was clear that he seemed to speak English, but it was not any easier. Sometimes some of the sounds of his speech led me to a distant thought about what he meant. But ultimately our communication was reduced to correspondence. I strained my memory, remembering all the best moments from Indian cinema and trying to make him feel the depth of our grief. As time went. India, as it should be, sang and cried, but could not help us either.
At 4 o'clock in the morning we realized that there was no other option but to recover from the backup. All eyes were on the tape library. Nicholas even crawled in front of her, wiping the floor with his tie. "How much will the base from the ribbon unwind?" I asked the local bakapschika. "Probably three hours," he answered uncertainly, "but we never checked." We urgently needed to replace the robot in the library, because we had three hours with great stretch. I have never so quickly opened the package and tightened the screws. With the bloodstained robot, it was finished, he fell into place. I filled it with blood, injuring my finger because of haste, but there was no time for dressing. He fluttered the toggle switch and the library was lit from the inside with light, the robot began to move and began to calibrate. We crowded around the library and looked through the window, mentally helping him move and read the barcodes from the tapes. He did it slowly, with pleasure, and we, having pressed against the glass, gnashed our teeth. “Initialize!” Said bakapshchik, referring to the library. And he added: "I launch the recovery." The robot thought for a second, then briskly darted to the right row of cassettes, confidently snatched the right one, turned around the guide axis and drove towards the drive.
At this time, Alex in the corner behind the boxes admired the film and another roll. The film went without sound, but the excitement on the face of my hapless colleague was genuine. Engineers often carry with them on long trips a selection of their favorite films that are good even without sound. For real art, such as movies with Charlie Chaplin, you can watch it like that.
Suddenly from the library I heard the sound of a dull thump. The robot took the cassette to drive, but could not determine where the drive had a hole, and began to beat against its wall, trying to insert it somewhere. It was a fiasco, and despair swept over you. Unfortunately, mechanical parts are extremely sensitive to transportation, so there are cases when a new part is immediately classified as DOA (dead on arrival).
Cold sweat rolled down my forehead and covered my eyes. I slowly pressed the keys, rechecking each. We had less than an hour before the start of the working day, and all our attempts already seemed in vain, when a thought struck me. “Have you been collecting diagnostics from the old server for a long time?” I asked my colleagues. "Last night, but what?" - answered me good-natured hungry Semen. "There may be a command output that reflects the configuration of the disk group before the changes, and according to this conclusion we can try to recreate it with our hands."
And now I was engaged in the fact that with my hands I ran commands that re-mark the disks into the service area and the data area. ” “Have you ever done this before?” Asked Nikolai, crawling from somewhere. “No, I only read the article, this is shamanism, but we must try,” I replied. Another time, I would immediately cut off my hands if they knew that I had just crashed the entire disk group. But now it was our only chance, so there were no objections. We all in this room were both condemned and conspirators. Hope in man never dies.
I finished with the markup and typed the command, which according to the documentation was to re-create the disk group, but did not initialize the space, i.e. Do not overwrite existing data. "Do - do not be afraid, afraid - do not," - I remembered Confucius and launched the team. She, to my surprise and joy, was fulfilled and did not produce a single error. I brought the configuration, all the logical volumes were in their places, with normal status. It remains to check the availability of data. In our stock was another 15 minutes. In the room already 40 minutes the phone rang, but nobody paid attention to it. Whoever it is, there is nothing to answer. “Let's run the cluster,” I suggested, and I was surprised at my voice.
In the library, the robot frantically circled about the drive, apparently hoping sooner or later to take it into a stupor. Alexey stroked the laptop, impressed with the ending of the film. Semyon patiently missed, thinking about whether he chose the right profession, and during breaks he dreamed of chocolates. Someone else was snoring behind the racks, now and then twitching in his sleep. The rest of the characters focused on the monitor, on which, without embellishment, our whole life was reflected. “It started!” Nikolai gasped out and grabbed the ringing tube. These were "applied", they already washed the ropes and called to say goodbye. “Check the butt!” Nikolai barked and brushed off his tie.
I sat on the box and smiled happily, thinking, as now, from all the dwelling houses of this beautiful city, people in pea jackets, earflaps and felt boots stretch in small streams. They are going to the queue at ATMs, and they also joyfully and reliably serve them. And in this whole picture there was some kind of right human happiness.
Everything worked for us, and 2 minutes before the start of the working day the system was started. Colleagues silently thanked me, shaking hands and shaking my head. Everything that happened that night, we were going to carry with us, not giving out to anyone.
I still continued to write a Hindu, still offering some crazy ideas. At this time, the door to the machine room opened and someone's sleepy face stuck out. It was a guard with a checkpoint. He said: “There the courier arrived. He says he brought some robot. Librarian robot like ... ".
Began a new wonderful working day.
This is how a tape storage looks (ours was smaller) and a library robot (ours is the same). Chinese not included.
Part 1. Box
I remember, there were the last days of November. Already thinking about the end of the working day, I was planning my evening, when suddenly I was informed that in the glorious Siberian city N our customer had a tape library broken. The spare part was immediately sent by the transport company, but for 3 days already, as it was still on the way. The transport company was vaguely explained and hmakala in the phone, and the customer was not in a joke nervous. Forecasts were uncertain, so it was decided to carry another spare part on their own by plane. A warehouse employee handed me a size box weighing ten kilograms, pasted over with bar codes and stickers, and happily slammed me on the shoulder with the words: "Just do not even think of taking my luggage - they will remember."
')
The box certainly disturbed me, but I was equally worried about a bag of dry milk, which at the last moment was handed to me by the project manager. “They have some troubles with milk now ... because of bad weather or something ... local guys asked them to bring 2 kg. Well you are not difficult? "- He said. From the expression of his face and the characteristic gesture of the palm, as if covering my mouth, it was clear how he now wanted me to be compliant or even dumb.
Important retreat
This story happened a few years ago, when I was an engineer, when I flew around the country a lot, and when I woke up in another hotel I sometimes couldn’t figure out which city I was in right now. 4-5 trips per month left their mark on the way of life. And the embodiment of this structure was my backpack, which was always ready for the next trip.
Part 2. Librarian
I got out of a taxi, and suddenly a police officer stopped me at the terminal building. “Your documents!” He spat out, and for some reason, a feeling of deepest satisfaction spread across his face. An evil dank wind blew, preventing me from sorting out my pockets and bags. I, as I could, took out my documents and handed them to the law enforcement officer. In theory, a young man who travels only with a laptop, a large box and a bag of white powder should not cause any suspicion, so I depicted a sincere bewilderment on my face. “What do you have in the box?” He asked indifferently. Apparently the fact that the package was nothing like dried milk, he had already identified with his professional sense, and therefore he was no longer interested in the package. But the box has not yet yielded to his insight. “This is a robot,” I said.
At that moment I had a feeling that his indifferent face was replaced, because it was lightened from the inside with genuine interest, as if a module with the function of human curiosity was additionally connected in the depths of his cap with a cockade. I understood that I was in a hurry with the answer and it was necessary to fix the situation somehow. "This is a robot ... for the library," I added even less confidently, continuing to monitor the connection of the remaining modules of malice, gloating, and greed. "Speak, the robot is for the library ... is it a librarian or something?"
Part 3. Inspection
I moved to the landing zone, remembering how I had just finished with a cockade. “It cost,” I thought, “and could have been late for the plane.”
“We remove luggage for tape, jackets, belts and shoes, lay out our phones ...” - a woman in uniform commanded powerfully and soullessly. Having mounted my belongings on the tape, I myself without sound and peep passed through the frame of the metal detector. It was clear that the box with the robot was haunting the airport security officer who drove her back and forth through an x-ray. “What is it?” She finally said when all the comparative images in her head had run out. I crushed a bag of milk in my hands, pondering my tactics so as not to repeat the cockade episode now. But suddenly her colleague, sitting next to her, glanced at the monitor and said: “This is a robot for the L200 library. For this week, the third time being taken, the season of breakdowns they went. Come on in, young man! ” There was no limit to my amazement, but at the same time I was terribly grateful to my providence, as well as to the technical literacy of fellow citizens working at airports.
Part 4. Manor
Rusty "Volga" creaked rushing me from the airport to the hotel. Of the thirty people of dubious kind of taxi drivers who met the night flight in front of the local airport building, this one seemed to me the most restrained. In all his slouching and smoky appearance, there was a sense of integrity and even nobility. However, the appearance of a taxi driver is often deceptive, and he wondered all the way why I live in Moscow, but I don’t know how Alla Borisovna Pugacheva is doing. “You live with her in the same city, and you don't know?”, My taxi driver did not let up. He disliked me for ignorance, but at the same time did not want to lose face with the face of the capital's guest. To the door of the hotel we drove famously and with a squeal of brakes, completing a steep pirouette, which is not allowed for an auto-assembly with such a venerable age. The door of the car seemed to open itself, or just fell off. Hearing my question: “Will you give me a check?”, The taxi driver waved his hands and completely forgot his conversational skills.
I went inside the building, looking around at the tastelessly decorated walls and pillars, all with some sort of gray bloom, like dust. The building was an old building, possibly a former manor. At the reception, the girl idly painted her nails. Confident in my own way as a client, I resolutely took out my passport and slapped it on the counter. Madame reluctantly distracted herself from her nails, looked contemptuously at my passport and muttered: "We do not give in debt." “What ...?” - interjections and other signs of extreme indignation ran in my head. “I have a room booked here,” I tried to be polite. “This is a casino! The entrance to the hotel is on the other side of the courtyard. ” Who knew that both places have the same name? And then I managed to think that the taxi driver had been driving me all this time, apparently thinking that I had flown from Moscow to this unwelcoming winter region to scour the contents of the box behind the cards and the tape measure.
Part 5. To sleep or not to sleep?
The hotel room met me with chic curtains to the floor and a musty smell of damp. I sat down on the edge of the bed, located in a niche of the wall, gently lowered the cherished box with the robot in front of me and greeted me unexpectedly with a spider in the corner. Works at the customer were scheduled for 9 am, the clock was 6 local. In my head, the IL-62 engines and other charms of the many hours flight were still buzzing. I mentally figured the prospects to sleep a couple of hours and wake up fresh, and made the right decision just to stay up. Physiologically, my case was headed for a late dinner, even a kefirchik for the night, so I straightened our corporate green tape on the box and went into the shower.
Part 6. Semyon
When by 9 am I arrived with a robot and powdered milk to the customer, at the entrance I was met by a representative of the IT department, who said that his name was supposedly Semen. Semyon talked about everything around, wiping his mouth with his hand every now and then. By his appearance and rustic manners, it was clear that in the department he meets the role more often than others, as the most illiterate specialist, but a very sociable person. Without a doubt, Semyon inspired confidence in people, and, as it turned out later, had a wonderful, almost indomitable appetite.
There was a fuss on the floor and it smelled of paint. Rapidly gesticulating, Semyon explained that this is a new building and now a second site is equipped here. And then my mobile phone rang. It was the project manager. “Have you got to the customer?” Listen, help them there. We are now setting up a cluster. Our Lesha has a week. Something does not work. Well you are not difficult?
Part 7. Cluster
Accompanied by Seeds, I entered the workshop. Everywhere there were boxes of different sizes labeled Dell, Cisco, EMC. Many of them had already been opened, around lay the wires, rails for mounting equipment in racks, bales of documentation and electricians. The electricians were not just lying, they were swarming with wires under the raised floor and were quietly quarreling. In the center of this creative chaos stood the senior specialist Nikolai. He held a workshop. Hiding from the meeting, our Alexey languished behind a pile of boxes, squinting at the monitor of his laptop and sipping breakfast on the sly. Semyon also wanted a loaf, but he had to participate in the meeting, so he just puffed.
From the first words of Nicholas, I realized that the launch time of the complex was running out, less than a day. With the advent of the second site, management has set the task of reorganizing one of the most important information systems. The database — the core of this information system — was supposed to be clustered. Now it was stored on two storage systems, between which synchronous data replication was carried out. And the cluster software tracked the state of the whole complex and could automatically respond to a failure, if something bad happened on the main site. The preparatory work was almost completed, but a lot of time was lost in the search and elimination of broken optical links between the sites that the contracting organization laid. And still had to finish the installation of servers, upgrade the operating system and microcodes, and test everything. The launch of the complex into commercial operation was scheduled for the night - the only time when it is possible to stop the responsible financial systems.
Part 8. From dawn to dusk
I don’t remember how this day went. I only remember that colleagues immediately declared to me that he would wait for the tape library in the light of the circumstances, and now my hands are more important in another matter.
The whole day we mounted servers, twisted nuts, laid cables, connected adapters. In the breaks, we even seemed to have lunch and went for a walk in the forest. Then I put the operating systems, rolled patches, conjured with microcodes. Sometimes microcodes conjured with me, and in order not to nap, I remembered how on a plane a flight attendant skillfully juggled a teapot, complaining about the turbulence, and distributed tea to wary boiling passengers. Night was falling, from the window it was clear that a large orange moon had risen above the forest.
Part 9. Hour X
“Hour X has come” - Nikolay said cheerfully, yawning, briefly conferring with someone on a local telephone. "It has come, it has come," they repeated to him from a pipe, which he had not yet had time to put in, and giggled. These were the “appliedists” who were also on duty that night and upon completion of our work had to make sure that everything worked.
On the clock, the hands showed midnight. We came to the crucial moment when we had to switch the replication of data to another storage system, wait for the synchronization to complete and raise the cluster at the second site. This process was commanded completely by Alexey, a well-known storage systems specialist in our circles. Some more were aware of his irrational habits ... He smacked and copied the volume identifiers into the script, referring to the list provided by the customer. Half an hour later, when this painstaking work was finished, Alex touched the enter key for fidelity, and making sure that it was real, pressed it. Everyone waited anxiously for something, and the general silence was broken only by the crackle of luminescent lamps. I tended to a raised floor and I had a dream in reality, but I do not remember what.
Part 10. Untitled
I woke up because the synchronization was over, but for some reason everyone was not happy. Cluster services did not rise. I was called from the country of dreams, so I looked in the logs. Through the fog in my mind it came to me that the problem was with the disk subsystem. The logical volume group was imported with an error and the data was inaccessible. The reasons were not yet clear, but the prospect of a rollback to the old configuration began to emerge; Plan b". At seven o'clock in the morning, the system had to be operational, so there was no time to think. Then I could not even imagine that the tragic events of that night were just beginning to unfold.
I meditated another five minutes and offered to roll back to the old system. Reluctantly, my colleagues agreed, dejectedly thinking that in the morning they would report to the authorities. We switched the replication direction and gave the command to import the disk group on the old server. I had already calmed down and dreamed of getting to the hotel bed with a sense of accomplishment. And suddenly I heard an exclamation: "And here it also does not start." And inside of us everything broke ...
Part 11. Akella missed
The fog finally disappeared in my head. It is said that in an extreme situation, the human body is mobilized. I was madly thinking what to do. The storage systems were fine. After confirming this, my colleague Alexey gave up his hand, signing his impotence, and went to watch a movie on a laptop. I have already spoken about his irrational actions. But formally, there was nothing to complain about.
The cause of the problems loomed quite clearly. Previously, additional volumes were added to the disk group by the customer; however, they were forgotten to be added to the list of disks that are replicated. When the cluster started, the disk group was imported, but the metadata was updated only on parts of the disks, while the new ones remained in read-only mode. Now the disk group configuration was finally mismatched, and the data is not available with either the new storage system or the old one.
When it comes to financial data of a huge organization, their loss is tantamount to a catastrophe. I immediately remembered a similar story when, at the dawn of my engineering career, I ended up with a repair from a single customer - a serious structure that was legalized in the 90s. They discussed that their admin unintentionally deleted something. “And what happened to him?” - among other things, I asked. “Well, he is still sitting here in the basement, the third day already. We are waiting for the general, he will decide what to do with him, ”the skinhead officer with a heavy chain on a bull neck answered me quite seriously.
Part 12. Hindi
Have you ever felt your hair moving on your head, going to turn gray? Pretending that we lost the database, I didn’t even think about what would happen in the morning.
In situations such as ours, it is necessary to immediately call the manufacturer’s technical support. It is, at such an hour, of course, English. I roughly sketched out keywords for myself and dialed a number. Two beeps and the English baritone answered me. I told immediately what we are and what happened to us. Baritone patiently listened to my monologue and asked: "What is the first name?". I was taken aback, but, having collected my thoughts, answered. Then questions about last name, customer name, city, location, serial number followed and so on to infinity. Time ran inexorably. After finishing the survey, the baritone asked me to record the number of the request and told me to wait for the call of a specialist. The specialist did not call me, he wrote me an e-mail in 5 minutes. He wrote that he accepted my application to work, but his shift was over, so I need to call the dispatcher again and ask him to assign an application to another specialist. It is not clear why some people believe that they do not like to work in Russia. It seems to be a delusion.
And suddenly a miracle happened. Another specialist called me back, but he was from India. The second global technical support center was there and they themselves accepted my application. I was glad to him as my own father, but my joy did not last long: we did not understand each other. It was clear that he seemed to speak English, but it was not any easier. Sometimes some of the sounds of his speech led me to a distant thought about what he meant. But ultimately our communication was reduced to correspondence. I strained my memory, remembering all the best moments from Indian cinema and trying to make him feel the depth of our grief. As time went. India, as it should be, sang and cried, but could not help us either.
Part 13. At dawn the penalty
At 4 o'clock in the morning we realized that there was no other option but to recover from the backup. All eyes were on the tape library. Nicholas even crawled in front of her, wiping the floor with his tie. "How much will the base from the ribbon unwind?" I asked the local bakapschika. "Probably three hours," he answered uncertainly, "but we never checked." We urgently needed to replace the robot in the library, because we had three hours with great stretch. I have never so quickly opened the package and tightened the screws. With the bloodstained robot, it was finished, he fell into place. I filled it with blood, injuring my finger because of haste, but there was no time for dressing. He fluttered the toggle switch and the library was lit from the inside with light, the robot began to move and began to calibrate. We crowded around the library and looked through the window, mentally helping him move and read the barcodes from the tapes. He did it slowly, with pleasure, and we, having pressed against the glass, gnashed our teeth. “Initialize!” Said bakapshchik, referring to the library. And he added: "I launch the recovery." The robot thought for a second, then briskly darted to the right row of cassettes, confidently snatched the right one, turned around the guide axis and drove towards the drive.
At this time, Alex in the corner behind the boxes admired the film and another roll. The film went without sound, but the excitement on the face of my hapless colleague was genuine. Engineers often carry with them on long trips a selection of their favorite films that are good even without sound. For real art, such as movies with Charlie Chaplin, you can watch it like that.
Suddenly from the library I heard the sound of a dull thump. The robot took the cassette to drive, but could not determine where the drive had a hole, and began to beat against its wall, trying to insert it somewhere. It was a fiasco, and despair swept over you. Unfortunately, mechanical parts are extremely sensitive to transportation, so there are cases when a new part is immediately classified as DOA (dead on arrival).
Part 14. Hope
Cold sweat rolled down my forehead and covered my eyes. I slowly pressed the keys, rechecking each. We had less than an hour before the start of the working day, and all our attempts already seemed in vain, when a thought struck me. “Have you been collecting diagnostics from the old server for a long time?” I asked my colleagues. "Last night, but what?" - answered me good-natured hungry Semen. "There may be a command output that reflects the configuration of the disk group before the changes, and according to this conclusion we can try to recreate it with our hands."
And now I was engaged in the fact that with my hands I ran commands that re-mark the disks into the service area and the data area. ” “Have you ever done this before?” Asked Nikolai, crawling from somewhere. “No, I only read the article, this is shamanism, but we must try,” I replied. Another time, I would immediately cut off my hands if they knew that I had just crashed the entire disk group. But now it was our only chance, so there were no objections. We all in this room were both condemned and conspirators. Hope in man never dies.
I finished with the markup and typed the command, which according to the documentation was to re-create the disk group, but did not initialize the space, i.e. Do not overwrite existing data. "Do - do not be afraid, afraid - do not," - I remembered Confucius and launched the team. She, to my surprise and joy, was fulfilled and did not produce a single error. I brought the configuration, all the logical volumes were in their places, with normal status. It remains to check the availability of data. In our stock was another 15 minutes. In the room already 40 minutes the phone rang, but nobody paid attention to it. Whoever it is, there is nothing to answer. “Let's run the cluster,” I suggested, and I was surprised at my voice.
In the library, the robot frantically circled about the drive, apparently hoping sooner or later to take it into a stupor. Alexey stroked the laptop, impressed with the ending of the film. Semyon patiently missed, thinking about whether he chose the right profession, and during breaks he dreamed of chocolates. Someone else was snoring behind the racks, now and then twitching in his sleep. The rest of the characters focused on the monitor, on which, without embellishment, our whole life was reflected. “It started!” Nikolai gasped out and grabbed the ringing tube. These were "applied", they already washed the ropes and called to say goodbye. “Check the butt!” Nikolai barked and brushed off his tie.
Part 15. Box
I sat on the box and smiled happily, thinking, as now, from all the dwelling houses of this beautiful city, people in pea jackets, earflaps and felt boots stretch in small streams. They are going to the queue at ATMs, and they also joyfully and reliably serve them. And in this whole picture there was some kind of right human happiness.
Everything worked for us, and 2 minutes before the start of the working day the system was started. Colleagues silently thanked me, shaking hands and shaking my head. Everything that happened that night, we were going to carry with us, not giving out to anyone.
I still continued to write a Hindu, still offering some crazy ideas. At this time, the door to the machine room opened and someone's sleepy face stuck out. It was a guard with a checkpoint. He said: “There the courier arrived. He says he brought some robot. Librarian robot like ... ".
Began a new wonderful working day.
Source: https://habr.com/ru/post/227341/
All Articles