Unix Recovery Legend
Now there are a lot of thoughts about what awaits us in the future, in 2009, and even after. But why not for a moment return to the past and admire how the hardcore unixoids of that time got out, restoring the system?
This is a translation of the article by Mario Wolczko published in Usenet in 1986.
Has it ever happened that you left a terminal logged in, just to go back and see how a (supposed) friend wrote rm -rf ~ / * in it and stands near the keyboard: - Lend me a five until Thursday, or I press enter .
')
Without a doubt, this person does not understand what kind of injury he can inflict, and perceives everything as a nice joke.
It was a quiet environment. To be precise - Wednesday October 1st, 3:15 pm British Daylight Time. Peter, my colleague, walked away from his terminal and told me: - Mario, I have a small problem with sending mail.
Understanding that such a message could confuse anyone, I decided to walk to his terminal to see what was wrong.
There was a strange error message in the terminal, something like this (I don’t remember all the details): cannot access / foo / bar for userid 147
At first I thought: who has userid 147? The sender of the message, the recipient or something else? Then I switched to another, already logged terminal, and dialed
just to get an answer
Immediately I suggested that something is not. All confirmed in response to
I got
I advised Peter that it would be a good idea not to touch anything now, and went to look for our system administrator.
When I came to his office, his door was ajar, and within 10 seconds I realized what the problem was. James, our manager, sat with his head in his hands, hands between his knees, like a man whose world had just collapsed. Our newly appointed system programmer, Neil, stood behind him and watched his terminal intently and sluggishly on his screen. And I looked at the top of the screen:
Oh, shit, I thought. And this explains everything.
I don’t even remember what happened next; my memory is blurred. I only remember that we tried ls (again), ps , who and maybe a few more commands - all is useless. The next thing I remember: I’m at my terminal again (multi-window graphic terminal), dialing
I have to thank David Korn, because he has made echo embedded inside the shell; Needless to say, the binary file / bin / echo for a company with all / bin has been removed. What became clear in the next few minutes is that / dev , / etc and
/ lib disappeared inseparably, but fortunately, Neil interrupted rm at a time when she was somewhere between / news and / tmp ; / usr and
/ users left untouched.
In the meantime, James got to our cabinet with cassettes and pulled out something with the inscription that it was a backup of the root file system, made four weeks ago. The question was in the air: - How can we restore the contents of the tape?
After all, we lost not only / etc / restore - all the device files of the tape drive controller were erased. Where does mknod live? Right, / etc. How about recovering any of them over Ethernet from another VAX? Understandably, / bin / tar was gone, and rcp people from Berkeley carefully put in
/ bin in the 4.3 distribution. In addition, for the network we need at least / etc / hosts . We found the cpio version in / usr / local , but, unfortunately, it is useless without a tape drive controller.
Alternatively, we could pull out the boot tape and rebuild the root filesystem, but neither James nor Neil had ever done this before, and we were not sure that this was what we needed - a completely reformatted disk and the loss of all our user files (we back up user files every Thursday; according to Murphy's law, everything should have happened on Wednesday).
Another solution is to borrow a disk from another VAX, boot from it, and only then figure it out; but then you would have to call a DEC-engineer - this is at best. We had many users in the throes of completing their PhD theses, and the loss of possibly weekly work was unthinkable.
So what to do? The next idea was to write a program that would create a device descriptor for a tape drive controller, but we all know where cc , as and ld live. Or, maybe, make the minimum view of / etc / passwd , / etc / hosts , etc., so that / usr / bin / ftp can work. By luck, I was still open in one of my gnuemacs windows - we could use it to create passwd and everything else, but the first step is to create a directory to put them there. Of course, / bin / mkdir was removed, the same thing happened with / bin / mv , so we could not rename / tmp to / etc. However, this was clearly the right line of attack.
Alasdair, our local UNIX guru, who turned out to be a VAX assembler, joined us by that time. So our plan was this:
It remains to run the program. Another miracle: the terminal, which was used to damage, was still superuser after su (just remember that su is in / bin ), so we at least have a chance that it will work.
And now, we are already on the charming path to success. Spending just an hour, we concocted about a dozen lines in assembler to create / etc. The cropped binary file was only 76 bytes long, so we converted it to HEX (reads a little better than the uuencode output), and recorded it using my editor. If you ever have such a problem, here is the HEX for the future:
I had an improvised program (and what, someone did not have?) To convert ASCII HEX to binary code, and the output of / usr / bin / sum coincided with our original binary file. But stop, wait a second - how do you set permissions to execute without / bin / chmod ? For a few seconds, a thought formed (which, as usual, completes a couple of minutes of thought) brought a solution to the problem: you need to write a binary file on top of an existing file for which I am the owner. That's good.
We returned to the terminal with superuser privileges, carefully remembered to set the umask to 0 (so that I could create files inside using my GNU), and started the binary file. Now we had / etc , with write access for everyone. Now there were only a few steps left: you had to create passwd , hosts , services , protocols , (etc), after which ftp was ready to go. Then we restored the contents of / bin over the network (it's incredible how terribly you miss ls after just a few short hours without it) and took the files from / etc. The key file was / etc / rrestore , with its help we restored / dev from backup tapes. Here and the fairy tale is over.
And now we ask ourselves the question: what is the moral of this story? Well, for starters, remember well the eternal words: do not panic . After all, at first we just wanted to restart the computer and try everything as a single-user, but, unfortunately, the system would not have found / etc / init and / bin / sh at startup. Common thinking saved us from such actions.
The next thing to remember is that the UNIX utilities can be used for purposes that are not really typical for them. Even without my gnuemacs, we could survive using, say, / usr / bin / grep as a replacement for / bin / cat .
And the last thing. It is unbelievable how much a huge part of the system can be removed without finally destroying it. Despite the fact that no one would be able to log in ( / bin / login ?), And almost all the necessary commands were gone, everything else looked fine. Naturally, some things cannot remain alive without, say, / etc / termcap , / dev / kmem or / etc / utmp , but, ultimately, everything works in peace and harmony.
I leave the question: being in the same situation, with the ability to think, given the experience acquired now, could this problem be solved more easily?
This is a translation of the article by Mario Wolczko published in Usenet in 1986.
Has it ever happened that you left a terminal logged in, just to go back and see how a (supposed) friend wrote rm -rf ~ / * in it and stands near the keyboard: - Lend me a five until Thursday, or I press enter .
')
Without a doubt, this person does not understand what kind of injury he can inflict, and perceives everything as a nice joke.
It was a quiet environment. To be precise - Wednesday October 1st, 3:15 pm British Daylight Time. Peter, my colleague, walked away from his terminal and told me: - Mario, I have a small problem with sending mail.
Understanding that such a message could confuse anyone, I decided to walk to his terminal to see what was wrong.
There was a strange error message in the terminal, something like this (I don’t remember all the details): cannot access / foo / bar for userid 147
At first I thought: who has userid 147? The sender of the message, the recipient or something else? Then I switched to another, already logged terminal, and dialed
grep 147 / etc / passwd
just to get an answer
/ etc / passwd: No such file or directory.
Immediately I suggested that something is not. All confirmed in response to
ls / etc
I got
ls: not found.
I advised Peter that it would be a good idea not to touch anything now, and went to look for our system administrator.
When I came to his office, his door was ajar, and within 10 seconds I realized what the problem was. James, our manager, sat with his head in his hands, hands between his knees, like a man whose world had just collapsed. Our newly appointed system programmer, Neil, stood behind him and watched his terminal intently and sluggishly on his screen. And I looked at the top of the screen:
# cd <br/> # rm -rf *
Oh, shit, I thought. And this explains everything.
I don’t even remember what happened next; my memory is blurred. I only remember that we tried ls (again), ps , who and maybe a few more commands - all is useless. The next thing I remember: I’m at my terminal again (multi-window graphic terminal), dialing
cd / <br/> echo *
I have to thank David Korn, because he has made echo embedded inside the shell; Needless to say, the binary file / bin / echo for a company with all / bin has been removed. What became clear in the next few minutes is that / dev , / etc and
/ lib disappeared inseparably, but fortunately, Neil interrupted rm at a time when she was somewhere between / news and / tmp ; / usr and
/ users left untouched.
In the meantime, James got to our cabinet with cassettes and pulled out something with the inscription that it was a backup of the root file system, made four weeks ago. The question was in the air: - How can we restore the contents of the tape?
After all, we lost not only / etc / restore - all the device files of the tape drive controller were erased. Where does mknod live? Right, / etc. How about recovering any of them over Ethernet from another VAX? Understandably, / bin / tar was gone, and rcp people from Berkeley carefully put in
/ bin in the 4.3 distribution. In addition, for the network we need at least / etc / hosts . We found the cpio version in / usr / local , but, unfortunately, it is useless without a tape drive controller.
Alternatively, we could pull out the boot tape and rebuild the root filesystem, but neither James nor Neil had ever done this before, and we were not sure that this was what we needed - a completely reformatted disk and the loss of all our user files (we back up user files every Thursday; according to Murphy's law, everything should have happened on Wednesday).
Another solution is to borrow a disk from another VAX, boot from it, and only then figure it out; but then you would have to call a DEC-engineer - this is at best. We had many users in the throes of completing their PhD theses, and the loss of possibly weekly work was unthinkable.
So what to do? The next idea was to write a program that would create a device descriptor for a tape drive controller, but we all know where cc , as and ld live. Or, maybe, make the minimum view of / etc / passwd , / etc / hosts , etc., so that / usr / bin / ftp can work. By luck, I was still open in one of my gnuemacs windows - we could use it to create passwd and everything else, but the first step is to create a directory to put them there. Of course, / bin / mkdir was removed, the same thing happened with / bin / mv , so we could not rename / tmp to / etc. However, this was clearly the right line of attack.
Alasdair, our local UNIX guru, who turned out to be a VAX assembler, joined us by that time. So our plan was this:
- write a program in assembler that could either rename / tmp to / etc or create / etc ;
- assemble it on another VAX, make uuencode ;
- write it to a uu-encoded file using my GNU, and make a uudecode (some clever person thought to put uudecode in / usr / bin ).
It remains to run the program. Another miracle: the terminal, which was used to damage, was still superuser after su (just remember that su is in / bin ), so we at least have a chance that it will work.
And now, we are already on the charming path to success. Spending just an hour, we concocted about a dozen lines in assembler to create / etc. The cropped binary file was only 76 bytes long, so we converted it to HEX (reads a little better than the uuencode output), and recorded it using my editor. If you ever have such a problem, here is the HEX for the future:
<br/> 070100002c0000000000000000000000 <br/> 00000000000000000000000000000000 <br/> dd8fff010000dd8f27000000fb02ef0700 <br/> 0000fb01ef070000000000bc8f88000400 <br/> 00bc012f65746300
I had an improvised program (and what, someone did not have?) To convert ASCII HEX to binary code, and the output of / usr / bin / sum coincided with our original binary file. But stop, wait a second - how do you set permissions to execute without / bin / chmod ? For a few seconds, a thought formed (which, as usual, completes a couple of minutes of thought) brought a solution to the problem: you need to write a binary file on top of an existing file for which I am the owner. That's good.
We returned to the terminal with superuser privileges, carefully remembered to set the umask to 0 (so that I could create files inside using my GNU), and started the binary file. Now we had / etc , with write access for everyone. Now there were only a few steps left: you had to create passwd , hosts , services , protocols , (etc), after which ftp was ready to go. Then we restored the contents of / bin over the network (it's incredible how terribly you miss ls after just a few short hours without it) and took the files from / etc. The key file was / etc / rrestore , with its help we restored / dev from backup tapes. Here and the fairy tale is over.
And now we ask ourselves the question: what is the moral of this story? Well, for starters, remember well the eternal words: do not panic . After all, at first we just wanted to restart the computer and try everything as a single-user, but, unfortunately, the system would not have found / etc / init and / bin / sh at startup. Common thinking saved us from such actions.
The next thing to remember is that the UNIX utilities can be used for purposes that are not really typical for them. Even without my gnuemacs, we could survive using, say, / usr / bin / grep as a replacement for / bin / cat .
And the last thing. It is unbelievable how much a huge part of the system can be removed without finally destroying it. Despite the fact that no one would be able to log in ( / bin / login ?), And almost all the necessary commands were gone, everything else looked fine. Naturally, some things cannot remain alive without, say, / etc / termcap , / dev / kmem or / etc / utmp , but, ultimately, everything works in peace and harmony.
I leave the question: being in the same situation, with the ability to think, given the experience acquired now, could this problem be solved more easily?
Source: https://habr.com/ru/post/47911/
All Articles