Erase can not be restored
Two interesting research articles from different parts of the world, published on the Web almost immediately after each other, give a significantly new look at the forensic aspects of SSD, or solid-state storage devices, often called flash drives.
The internal mechanisms of SSD work are so significantly different from traditional hard disk drives that forensic experts can no longer rely on current data storage technologies in situations where evidence from SSD-type media appears in court proceedings.
On the other hand, fragments of data stored in the memory of flash drives can be virtually indestructible.
Can not be restored
Approximately this is the essence of the warning in the results of a research article by scientists from the Australian University of Murdoch ("The State Forensic Discovery" by Graeme B. Bell and Richard Boddington, PDF).
')
The study was based on a large series of experiments comparing the nuances of data storage in the samples under study: a flash drive of a Corsair 64GB SSD and a traditional Hitachi 80GB magnetic disk. In a comparative analysis, the researchers found in SSD a whole bunch of problems with data recovery. Problems that are completely unrelated to magnetic disks and caused by cleaning or “garbage collection” algorithms used to keep flash drives at maximum performance.
Under the influence of these algorithms, the data important for the investigation, stored on modern SSDs, often become the object of a process that has received the name “self-corrosion” among forensic scientists. The result of this process is that the evidence on the SSD is continuously erased or contaminated with extraneous data - in a way that is completely uncharacteristic of carriers based on hard magnetic disks. And, which is fundamentally important, all these changes with information occur in the absence of any commands from the user or from the computer.
The results of Australian researchers inevitably raise doubts about the integrity and reliability of those files that are isolated by forensic methods and removed from storage devices. One could even say that there was a clear threat of the end of that “golden age” in the collection of digital evidence, which was provided by the data storage features on magnetic media.
Over the past few decades, investigators have worked with magnetic tapes, floppy and hard disks, which consistently continued to store huge amounts of information after the files that contained all of this were marked by the system as deleted. Even the procedure of safe stripping (wiping), as is known to specialists, is not always enough to completely destroy information on magnetic media. However, in SSD solid-state disks, data is stored significantly differently - in the form of blocks or pages of NAND transistor chips, which must be erased electronically before they can be reused.
The result of the work of the industry to improve the efficiency of SSD memory is that most modern flash drives have firmware built into the firmware that regularly and automatically performs “self-cleaning” or “garbage collection” procedures. As a result of these sanitary procedures, constant rubbing, modification and transfer of those files that are marked as destroyed by the system occur. Moreover, this process begins without any notice and very quickly, almost immediately after the power supply to the chip. No commands are required from the user, and flash drive does not emit any sound or light signals in order to inform the user about the start of the cleaning procedure.
When testing a specific sample, after it was subjected to quick formatting, the researchers expected the cleaning utility to start working in about 30-60 minutes, believing that this process should occur with the SSD before new data will be written to the blocks before occupied by files. To their surprise, the sweep took place only three minutes later, after which only 1064 files of evidence from a total of 316,666 remained available for recovery from the disk.
Having decided to follow this process further, the scientists removed the flash disk from the computer and connected it to a recording blocker — a hardware device specifically designed to isolate from all procedures that could change the contents of the media. But here, just 20 minutes after connecting, almost 19 percent of all files were overwritten due to internal processes that the SSD itself initiates without any external commands. For comparison, it can be noted that on an equivalent magnetic hard disk, all data after similar formatting remained recoverable, regardless of the elapsed time - as expected by the researchers.
It is clear that for criminologists who are concerned about the safety of all data on the media, this feature of the SSD is a big problem. As one of the co-authors, Graham Bell, writes in a commentary on their article, “several people in the computer forensics community had an idea that some funny things were happening with the data in the SSD, but almost everyone we showed our results was shocked the scale of what was discovered. "
If “garbage collection” in the SSD occurs before or during the forensic image removal procedure, then this leads to the irreversible destruction of potentially large arrays of valuable data. Those data that would normally have been obtained as evidence in the course of the investigative process, from which the new term was born - “corrosion of evidence”.
There is no doubt that the discovery of Australian specialists will inevitably have serious consequences for those criminal and civil court cases that rely on digital evidence. If the disc from which the evidence was obtained has indications that changes occurred with the data after the device was taken from the owner, then the opposing party will have reason to demand that this evidence be excluded from judicial review.
The authors of the article also warn that as the capacity of USB flash drives grows, manufacturers can begin to embed similar cleaning technologies in them, causing the same problem for an array of secondary (external) storage media. In addition, Bell and Boddington suggest that the garbage collection utilities will become more and more aggressive over time, as manufacturers introduce more and more powerful firmware in their functionality, chipsets, and larger disks.
In the final conclusion of the article containing 18 points of the problem, the researchers do not propose any treatment methods, believing that there is no simple and effective solution to this problem.
Can not be erased
If we talk about another American research article, also devoted to the specific features of data storage in the SSD, then at first glance its results seem to be clearly conflicting with those obtained by the Australians. Here, a team of researchers has come to a completely different discovery: fragments of data stored in the memory of flash drives may turn out to be virtually indestructible.
As the authors of this article demonstrate, flash drives are very difficult to clear of compromise-sensitive data using traditional methods of mashing files and disks safely. Even in cases where SSD devices show that files have been deleted, up to 75 percent of the data they contained may still be in flash drive memory. In particular, in some cases, when solid-state disks indicate that the files are “safely erased”, in fact, their duplicates remain largely intact in secondary locations.
These are, in brief, the findings of a study conducted at the University of California San Diego and presented in the last days of February at the Usenix FAST 11 conference ("Reliably Erasing Data From Flash-Based Solid State Drives" by Michael Wei, Laura Grupp, Frederick Spada, Steven Swanson PDF)
Problems with reliable rubbing of data on the SSD, as the authors of the work write, occur due to a radically different internal design of the carrier. Traditional ATA and SCSI drives use magnetized materials to write information to a specific physical location known as an LBA or logical unit address. SSD drives, on the other hand, use digital storage chips for managing content using FTL or “flash stream layer”. When data in such media is modified, FTL often writes new files to different places, simultaneously updating the memory map to reflect the changes made. The result of such manipulations is that the remnants of the previous files, which the authors call “digital remains”, in the form of uncontrolled duplicates continue to be stored on disk.
As the authors write, “these differences in processing between magnetic disks and SSDs potentially lead to dangerous discrepancies between user expectations and the actual behavior of the flash drive ... The owner of such a device can apply standard hard disk cleaning tools to the SSD, mistakenly believing that the data on the disk will be irreversibly destroyed. In fact, these data can remain on the disk and require only a few more complex operations to restore them. ”
If we talk about specific numbers, the researchers found that about 67 percent of the data stored in the file remained on the disk even after it was destroyed in the SSD using the "Safe Erase" feature available in Apple Mac OS X. Other Safe (overwritten) erase utilities with other operating systems showed similar results. For example, after the destruction of individual files by the Pseudorandom Data program on the SSD, up to 75 percent of the data could remain, while using the British government stripping technology British HMG IS5, up to 58 percent remained.
As the article warns, these results indicate: in the situation with SSD, overwriting data is ineffective, and the standard erasure procedures provided by manufacturers may not work properly.
According to researchers, the most effective way to safely delete data in SSD is the use of devices that encrypt their contents. Here, the Wiping procedure is reduced to the destruction of the encryption keys in a special section called the “keystore”, essentially ensuring that the data remains on the disk forever encrypted.
But here, of course, another problem lurks. As the authors of the article write, “the danger lies in the fact that the protection relies on the correct operation of the controller that cleans the internal storage compartment, which contains the cryptokey, and any other values ​​derived from it that may be useful in cryptanalysis. Taking into account the implementation errors that we found in some versions of the secure erase utilities, it would be unjustifiably optimistic to mean that SSD providers will clean up the keystore correctly. Worse, there is no way (for example, by disassembling the device) to make sure that the erasure really happened. ”
The researchers obtained their results by writing different files with well identifiable data structures on SSDs. After that, a special device based on FPGA (chips with reprogrammable logic) was used to quickly find and identify the remaining “fingerprints” of these files after applying the secure erase procedures. The special equipment of researchers costs about a thousand dollars, however, “a simpler version of the device based on a microcontroller would cost about $ 200 and would only require modest technical experience to design it.”
No contradiction
As the cumulative results of these two articles were formulated on the Slashdot discussion forum, “either SSDs are really hard to clean up, or they are really very difficult to recover deleted files from. It turns out some tangled story "...
One of the direct participants in the first (Australian) study, Graham Bell, explains this seeming paradox as follows.
Before, the data on the disks was traditionally cleaned manually, that is, giving the computer an explicit command, so that it would tell the drive to write something else over the previous data. If there was no such command to overwrite, then in magnetic media the data continued to be saved. However, if the same trick to try to apply to the SSD, then it may not work. That logical memory address that you are trying to overwrite could have already been reallocated on the fly, so your “overwrite” command goes to some other physical memory cell, and not to the one that stored the data before. From a logical point of view, it all looks like the rewriting worked: you can no longer get access to this data through the OS of your computer. However, from the point of view of the flash drive itself, this data is still there, hidden in some kind of physical cell, which is currently not used, if we mean the corresponding logical sector. However, some ingenious firmware or cunning hacker with a soldering iron, in principle, can get to this data.
At the same time, apart from these features, modern SSD carriers use different specific tricks in order to automatically increase their performance. One of these tricks is to overwrite the memory cells in advance, which contain data that is no longer considered by the file system. In this case, the drive itself actively tries to continuously clean everything that it can from the disk. And it does all this solely on its own initiative - just to speed up future write operations, providing a pre-prepared pool of accessible and unused cells.
Summarizing these features of SSD, we can state the following. If your computer tells the flash drive to reset some data, then the drive may lie to you, and in fact zeroing may or may not. If the drive itself wants to wipe something (and it actually does it without any warning), then this data will be destroyed ...
Another commentator, obviously not without a sense of humor, described such an intricate situation with these words:
“Why do you call it a confusion? Here everything is transparent and clear. If you want to recover the deleted data, then you can not do this. If you want to destroy them, then you can not do it. This is such a Murphy Law for storing data on an SSD. ”
Ps. The original is here http://www.computerra.ru/597770/ , unfortunately I can not publish a topic link, but the topic is extremely interesting, forgive me.
The internal mechanisms of SSD work are so significantly different from traditional hard disk drives that forensic experts can no longer rely on current data storage technologies in situations where evidence from SSD-type media appears in court proceedings.
On the other hand, fragments of data stored in the memory of flash drives can be virtually indestructible.
Can not be restored
Approximately this is the essence of the warning in the results of a research article by scientists from the Australian University of Murdoch ("The State Forensic Discovery" by Graeme B. Bell and Richard Boddington, PDF).
')
The study was based on a large series of experiments comparing the nuances of data storage in the samples under study: a flash drive of a Corsair 64GB SSD and a traditional Hitachi 80GB magnetic disk. In a comparative analysis, the researchers found in SSD a whole bunch of problems with data recovery. Problems that are completely unrelated to magnetic disks and caused by cleaning or “garbage collection” algorithms used to keep flash drives at maximum performance.
Under the influence of these algorithms, the data important for the investigation, stored on modern SSDs, often become the object of a process that has received the name “self-corrosion” among forensic scientists. The result of this process is that the evidence on the SSD is continuously erased or contaminated with extraneous data - in a way that is completely uncharacteristic of carriers based on hard magnetic disks. And, which is fundamentally important, all these changes with information occur in the absence of any commands from the user or from the computer.
The results of Australian researchers inevitably raise doubts about the integrity and reliability of those files that are isolated by forensic methods and removed from storage devices. One could even say that there was a clear threat of the end of that “golden age” in the collection of digital evidence, which was provided by the data storage features on magnetic media.
Over the past few decades, investigators have worked with magnetic tapes, floppy and hard disks, which consistently continued to store huge amounts of information after the files that contained all of this were marked by the system as deleted. Even the procedure of safe stripping (wiping), as is known to specialists, is not always enough to completely destroy information on magnetic media. However, in SSD solid-state disks, data is stored significantly differently - in the form of blocks or pages of NAND transistor chips, which must be erased electronically before they can be reused.
The result of the work of the industry to improve the efficiency of SSD memory is that most modern flash drives have firmware built into the firmware that regularly and automatically performs “self-cleaning” or “garbage collection” procedures. As a result of these sanitary procedures, constant rubbing, modification and transfer of those files that are marked as destroyed by the system occur. Moreover, this process begins without any notice and very quickly, almost immediately after the power supply to the chip. No commands are required from the user, and flash drive does not emit any sound or light signals in order to inform the user about the start of the cleaning procedure.
When testing a specific sample, after it was subjected to quick formatting, the researchers expected the cleaning utility to start working in about 30-60 minutes, believing that this process should occur with the SSD before new data will be written to the blocks before occupied by files. To their surprise, the sweep took place only three minutes later, after which only 1064 files of evidence from a total of 316,666 remained available for recovery from the disk.
Having decided to follow this process further, the scientists removed the flash disk from the computer and connected it to a recording blocker — a hardware device specifically designed to isolate from all procedures that could change the contents of the media. But here, just 20 minutes after connecting, almost 19 percent of all files were overwritten due to internal processes that the SSD itself initiates without any external commands. For comparison, it can be noted that on an equivalent magnetic hard disk, all data after similar formatting remained recoverable, regardless of the elapsed time - as expected by the researchers.
It is clear that for criminologists who are concerned about the safety of all data on the media, this feature of the SSD is a big problem. As one of the co-authors, Graham Bell, writes in a commentary on their article, “several people in the computer forensics community had an idea that some funny things were happening with the data in the SSD, but almost everyone we showed our results was shocked the scale of what was discovered. "
If “garbage collection” in the SSD occurs before or during the forensic image removal procedure, then this leads to the irreversible destruction of potentially large arrays of valuable data. Those data that would normally have been obtained as evidence in the course of the investigative process, from which the new term was born - “corrosion of evidence”.
There is no doubt that the discovery of Australian specialists will inevitably have serious consequences for those criminal and civil court cases that rely on digital evidence. If the disc from which the evidence was obtained has indications that changes occurred with the data after the device was taken from the owner, then the opposing party will have reason to demand that this evidence be excluded from judicial review.
The authors of the article also warn that as the capacity of USB flash drives grows, manufacturers can begin to embed similar cleaning technologies in them, causing the same problem for an array of secondary (external) storage media. In addition, Bell and Boddington suggest that the garbage collection utilities will become more and more aggressive over time, as manufacturers introduce more and more powerful firmware in their functionality, chipsets, and larger disks.
In the final conclusion of the article containing 18 points of the problem, the researchers do not propose any treatment methods, believing that there is no simple and effective solution to this problem.
Can not be erased
If we talk about another American research article, also devoted to the specific features of data storage in the SSD, then at first glance its results seem to be clearly conflicting with those obtained by the Australians. Here, a team of researchers has come to a completely different discovery: fragments of data stored in the memory of flash drives may turn out to be virtually indestructible.
As the authors of this article demonstrate, flash drives are very difficult to clear of compromise-sensitive data using traditional methods of mashing files and disks safely. Even in cases where SSD devices show that files have been deleted, up to 75 percent of the data they contained may still be in flash drive memory. In particular, in some cases, when solid-state disks indicate that the files are “safely erased”, in fact, their duplicates remain largely intact in secondary locations.
These are, in brief, the findings of a study conducted at the University of California San Diego and presented in the last days of February at the Usenix FAST 11 conference ("Reliably Erasing Data From Flash-Based Solid State Drives" by Michael Wei, Laura Grupp, Frederick Spada, Steven Swanson PDF)
Problems with reliable rubbing of data on the SSD, as the authors of the work write, occur due to a radically different internal design of the carrier. Traditional ATA and SCSI drives use magnetized materials to write information to a specific physical location known as an LBA or logical unit address. SSD drives, on the other hand, use digital storage chips for managing content using FTL or “flash stream layer”. When data in such media is modified, FTL often writes new files to different places, simultaneously updating the memory map to reflect the changes made. The result of such manipulations is that the remnants of the previous files, which the authors call “digital remains”, in the form of uncontrolled duplicates continue to be stored on disk.
As the authors write, “these differences in processing between magnetic disks and SSDs potentially lead to dangerous discrepancies between user expectations and the actual behavior of the flash drive ... The owner of such a device can apply standard hard disk cleaning tools to the SSD, mistakenly believing that the data on the disk will be irreversibly destroyed. In fact, these data can remain on the disk and require only a few more complex operations to restore them. ”
If we talk about specific numbers, the researchers found that about 67 percent of the data stored in the file remained on the disk even after it was destroyed in the SSD using the "Safe Erase" feature available in Apple Mac OS X. Other Safe (overwritten) erase utilities with other operating systems showed similar results. For example, after the destruction of individual files by the Pseudorandom Data program on the SSD, up to 75 percent of the data could remain, while using the British government stripping technology British HMG IS5, up to 58 percent remained.
As the article warns, these results indicate: in the situation with SSD, overwriting data is ineffective, and the standard erasure procedures provided by manufacturers may not work properly.
According to researchers, the most effective way to safely delete data in SSD is the use of devices that encrypt their contents. Here, the Wiping procedure is reduced to the destruction of the encryption keys in a special section called the “keystore”, essentially ensuring that the data remains on the disk forever encrypted.
But here, of course, another problem lurks. As the authors of the article write, “the danger lies in the fact that the protection relies on the correct operation of the controller that cleans the internal storage compartment, which contains the cryptokey, and any other values ​​derived from it that may be useful in cryptanalysis. Taking into account the implementation errors that we found in some versions of the secure erase utilities, it would be unjustifiably optimistic to mean that SSD providers will clean up the keystore correctly. Worse, there is no way (for example, by disassembling the device) to make sure that the erasure really happened. ”
The researchers obtained their results by writing different files with well identifiable data structures on SSDs. After that, a special device based on FPGA (chips with reprogrammable logic) was used to quickly find and identify the remaining “fingerprints” of these files after applying the secure erase procedures. The special equipment of researchers costs about a thousand dollars, however, “a simpler version of the device based on a microcontroller would cost about $ 200 and would only require modest technical experience to design it.”
No contradiction
As the cumulative results of these two articles were formulated on the Slashdot discussion forum, “either SSDs are really hard to clean up, or they are really very difficult to recover deleted files from. It turns out some tangled story "...
One of the direct participants in the first (Australian) study, Graham Bell, explains this seeming paradox as follows.
Before, the data on the disks was traditionally cleaned manually, that is, giving the computer an explicit command, so that it would tell the drive to write something else over the previous data. If there was no such command to overwrite, then in magnetic media the data continued to be saved. However, if the same trick to try to apply to the SSD, then it may not work. That logical memory address that you are trying to overwrite could have already been reallocated on the fly, so your “overwrite” command goes to some other physical memory cell, and not to the one that stored the data before. From a logical point of view, it all looks like the rewriting worked: you can no longer get access to this data through the OS of your computer. However, from the point of view of the flash drive itself, this data is still there, hidden in some kind of physical cell, which is currently not used, if we mean the corresponding logical sector. However, some ingenious firmware or cunning hacker with a soldering iron, in principle, can get to this data.
At the same time, apart from these features, modern SSD carriers use different specific tricks in order to automatically increase their performance. One of these tricks is to overwrite the memory cells in advance, which contain data that is no longer considered by the file system. In this case, the drive itself actively tries to continuously clean everything that it can from the disk. And it does all this solely on its own initiative - just to speed up future write operations, providing a pre-prepared pool of accessible and unused cells.
Summarizing these features of SSD, we can state the following. If your computer tells the flash drive to reset some data, then the drive may lie to you, and in fact zeroing may or may not. If the drive itself wants to wipe something (and it actually does it without any warning), then this data will be destroyed ...
Another commentator, obviously not without a sense of humor, described such an intricate situation with these words:
“Why do you call it a confusion? Here everything is transparent and clear. If you want to recover the deleted data, then you can not do this. If you want to destroy them, then you can not do it. This is such a Murphy Law for storing data on an SSD. ”
Ps. The original is here http://www.computerra.ru/597770/ , unfortunately I can not publish a topic link, but the topic is extremely interesting, forgive me.
Source: https://habr.com/ru/post/115349/
All Articles