How not to recover data, or so lucky for you too

We all occasionally encounter storage device failures. On the Internet, written hundreds of instructions on how to read without any special equipment everything that is possible from devices that still respond to normal OS requests. But I had no luck for many years, the disks either died very, very completely, or the file system was still available and I just read everything that was read in the usual way. And waited. Should it happen that the dying disk hit me in a state that requires more than the most basic actions?



And this day has come. On the section above the software raid0 , which, frankly, kept all the nonsense, the files were inaccessible, and the reason was the long-awaited I / O Error. Unexpectedly, the device of the carrier of the second half of Reid did not appear at all in the fdisk output, although the list of files was available. Reboot - and the array is assembled, and the files are perfectly readable. The SMART report shows a huge number of reassigned sectors. Just what you need. Turning off the drives and starting to think about a recovery strategy.



To read out the surviving sectors, it was decided to use GNU ddrescue - a specialized version of dd for reading data from dying disks, which allows you to choose a reading strategy and knows how to build a disk card to continue working after failures, but they will definitely work with the faulty device.



So, I connect the first disk to my desktop, and POST solemnly reports that one of the connected disks is completely bad. The OS boot was normal, except for a couple of messages about some ata device errors. After successful loading of fdisk -l, the patient sees, well, finally I will restore the integrity of the raid0 data.

')

ddrescue -ndAvv /dev/sdd1 bad mapbad 

The process went pretty quickly, 80 MB / s. Due to the noise of the cooling fan discs, specially removed on this occasion from the decommissioned server, the chilling sounds of reading a dying disk would not be heard, even if they were. Perhaps not hearing them is even better. The less you know the better you sleep.



By morning, the work of ddrescue was completed, and on the new disk lay the image and map of the perishing comrade. But for some reason, restarting ddrescue ended suspiciously quickly, and the map





“Something obviously went wrong,” suggested a sleepy inner voice. “Try fdisk -l ” - here, and the mind has already connected. / dev / sdd was not in the output. - How is this possible at all ?, - Oh, yes, there was the same thing on the server, - So, not everything was lost: approximately such thoughts lived in the head during the whole working day.



In the evening, the experimental disk again showed signs of life, and I had a difficult task: what to do with the image obtained, because ddrescue with a finalized map refused to read anything, leaving me with an image file smaller than the partition size.

On the one hand, ddrescue can fill - blocks with arbitrary data, in order to later identify broken files. On the other hand, I didn’t ask him to do a spars file (the -S parameter), and he’ve been reinforcing the question of how exactly the smaller-than-section image file will be filled in, remained unanswered.





I did not want to rely on the case, and I decided to study a disk card with a calculator in my hands. It turned out that 96% of the disk content was successfully read and got into the first + block. But unfortunately, the sum of all the + blocks did not give the size of the read image, which finally confused me all the cards. That's why I went the other way: since I have a good beginning and God knows what is in the end, why not get the good old dd and blind from an incomprehensible image exactly what you can safely work with?

 dd if=bad of=bad_new bs=16M count=42751 dd if=/dev/zero of=bad_new bs=1024 seek=700432384 count=32139617 

The result is a brand new image that is now exactly the same size as the dying section, namely 750153729024 bytes. Now let's do zeroes at the end. I do by hand





and run:

 ddrescue -ndAvv /dev/sdd1 bad_new mapbad 

Hooray! Now everything is fine. The read blocks from the disk moved to an empty space of the new image at the necessary addresses, and the map was again finalized, and my adventures are coming to an end. But maybe make the last spurt and re - read - blocks in one sector? Now give ddrescue





and run like this:

 ddrescue -ndAvv /dev/sdd1 -1 bad_new mapbad 

In general, he behaved not at all as I expected. Having bumped into a bad sector in the - block, he moved on to the next - block, until he caught on sure reading from the last - block. An hour later, I stopped this mess with CTRL-C and started reading at normal speed. A few more gigabytes from the end of the section were read until ddrescue again ran into bad sectors, which for some reason again led to the disappearance of / dev / sdd from the system and the finalization of the map, which was very swollen and was a mess of + , - and ? blocks. There was no point in analyzing all this, and I decided to stop tormenting the dying man and finally work on assembling the array. But before that I wanted to find out what it would be like if I used the fill mode. I prepare the placeholder and turn on the fill mode, not forgetting to replace the card with the original one:

 printf "BADSECTOR" > tmpfile ddrescue --fill-mode=- tmpfile bad mapbad 

And I get another image of the correct size! How he did it remained a mystery to me, because I repeatedly counted the sum of the sizes of the + blocks from the map, and it did not coincide with the size of the finished file.



And here, in the place of giving the magnetic soul to God, is already his healthy colleague, and I fulfill the commands:

 losetup /dev/loop1 bad_new mdadm --assemble -o /dev/md12 /dev/loop1 /dev/sdd1 

But the result, to put it mildly, is not what I expected after 24 hours of hope for success:

 mdadm: no RAID superblock on /dev/loop1 mdadm: /dev/loop1 has no superblock - assembly aborted 

What to do, we will try to communicate now with Google and sections:

 #mdadm --examine /dev/loop1 mdadm: No md superblock detected on /dev/loop1. 

- That is, how is it no md superblock detected ?

- And this one, which is still healthy?

 #mdadm --examine /dev/sdd1 /dev/sdd1: Magic : a92b4efc Version : 0.90.00 UUID : 64d38efd:b8e92c8c:f5292846:21864477 Creation Time : Fri Oct 10 20:22:23 2008 Raid Level : raid0 Raid Devices : 2 Total Devices : 2 Preferred Minor : 2 Update Time : Fri Oct 10 20:22:23 2008 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Checksum : 6f768a67 - correct Events : 1 Chunk Size : 4K 

- Version 0.90.00? And where should the mdadm 0.90.00 superblock be?

- At the end of the section.

- And how to read it with the help of mdadm ?

- None.

- So how is it anyway? Well, okay, so where is it more accurate?

- Not farther than 128K but no closer than 64K from the end of the section, aligned on the 64K border, 4K size.



N-yes, I take dd and check:

 dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128 dd if=last128 of=bad_new bs=1024 seek=732571873 count=128 

Yes, the superblock enrolled at the end of the section at the correct address, but it still doesn’t want to assemble an array with two identical superblocks, which means you need to give it exactly that superblock, with dying.



Replacing the disk again, just to read:

 #dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128 dd: reading `/dev/sdd1': Input/output error 0+0 records in 0+0 records out 

“Yes, the Great Combinator never experienced such a blow,” for some reason this immortal quotation came to mind.

Well, so easily, we do not give up. And we like this:

 dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128 conv=noerror 

Oh, the size of the last128 > 0 but <128K. And where is that? How to work with it? And how to make a zero instead of the dead sectors? That's how:

 dd if=/dev/sdd1 of=last128 bs=1024 skip=732571873 count=128 conv=noerror,sync 

Here they are, the last 128K. I register them to the end of the image, reboot with a healthy disk, and the array is assembled!

There is a simple command and:

 #mount -o ro /dev/md11 /srv/old/ mount: wrong fs type, bad option, bad superblock on /dev/md11, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so 

Well, this is somehow completely offensive. The last time the array was collected on the server and ext3 was mounted. The output of dumpe2fs does not cause suspicions, the status of the clean section, the addresses of alternative superblocks are visible, but for some reason they are not mounted with them either. It would be necessary to treat it, but in this form and with half the data without backup can not be treated. Oh, you have to throw another half a terabyte. Of course, in an amicable way, it would be necessary to make an image of the now conditionally working disk with the help of ddrescue , but all this is already very tired and therefore I do this:

 dd if=/dev/md11 of=ext3 bs=16M conv=noerror,sync 

And sleep, and in the morning I try:

 losetup /dev/loop2 ext3 mount -o ro /dev/loop2 /srv/old/ 

And it is mounted! At first glance, the directory structure has not changed, and the files from which I have hashes were read without errors. On the restored partition, only 20 GB is free, so its loop file received a well-deserved attribute:

 chattr +i ext3 

and write to fstab:

 /srv/ext3 /srv/old ext3 ro,loop 0 0 

As a conclusion, read up to here I recommend:

• configure automatic backup of important and not very data,
• periodically view SMART reports
• use a USB to SATA device or similar for your drive to restore,
• treat any disk from which you want to remove an image as if you were a dying person and use GNU ddrescue ,
• remember that if it comes to the recovery of really important data, then such luck will not be exact.