From backup to backup: the history of IT companies that have lost data

Most companies understand the importance of creating backups. But the trouble is that not many companies have an idea of ​​what the data backup strategy should represent. As a result, they lose information, customers, and therefore money. Back in 2014, experts reported that the business was losing about $ 1.7 trillion dollars a year due to irretrievable loss of the most valuable data, which for some reason were not reserved. Now this figure has grown, since the time-consuming idle data center costs the operator $ 50,000 - $ 80,000. Two years ago, an hourly simple entailed losses of $ 40,000 - $ 60,000.

Each year, this amount grows as the value of the data gradually increases. And not only in the information business - after all, the usual simple equipment because of all sorts of problems hurts the purse of the company itself, which owns the infrastructure, and its customers.

Only in the first half of 2016, 554 million records were lost or stolen as a result of cyber attacks by cybercriminals. The most common purpose of hacking is to steal users' personal data. In the US, healthcare was the most attacked area. At the same time, government agencies lost the maximum amount of data in the first half of 2016 (we are talking about lost or stolen data).
')
At the same time, for the entire 2015, 707.5 million records were lost or stolen as a result of cyber attacks. This, however, is less than in 2014, when the corresponding figure was 1.02 billion records.

Among the causes of data loss are companies' lack of readiness for critical events (power outages, physical damage to equipment, hacking and data theft, natural disasters). If you do not take care of backups in advance - then it will be painfully painful, many have heard about it. But, as they say, "the mice cried, injected, but continued to eat the cactus." Let's take a look at some unexpected cases that suddenly led to a partial or complete shutdown of the telecommunications companies and their divisions. This is not always about data loss and backup, but these situations make you think about how quickly a well-established process / work of a company can turn into chaos. Despite the planning, unexpected situations may arise, and necessarily arise, sooner or later.

Fun toy story

One of the most notorious stories earlier is that Pixar lost a large array of data on ToyStory 2. Then one of the employees accidentally erased from the database server hundreds of important elements of character animation, the sources of the characters themselves, etc. After the company decided to restore the data from the backup, it turned out that the backup had not worked for more than a month.

There was a threat that a whole month of work (or even more) went nowhere. But then it turned out that one of the project managers regularly sent all the data to his home PC in order to be able to work on the project at home. Only due to this (and, it should be noted, it was a violation of corporate rules), we managed to restore the data.

If the data did not appear on the home server, the timing of the project could drag on, and the company would be in a very unpleasant situation.

Accident involving data center

If we talk about sudden events that led to loss and loss of data, here it is impossible not to recall the incident from 2007. Then the company Rackspace (which was not yet as authoritative as years later) came across a surprise. An SUV crashed into its data center. The driver of this car suffered from diabetes. During the trip, he lost consciousness, his foot pressed the gas pedal, and the car, flying out of the roadway, crashed at full speed into the facility where the center of the company's data center energy infrastructure was located.

Immediately earned auxiliary power supply system, but a problem arose - the main cooling system did not start. Because of this, the equipment quickly overheated, so that employees of the company decided to turn off everything so that servers and other equipment would not fail.

As a result, the data center stood idle for about five hours, during which nothing worked. These five hours cost the company $ 3.5 million. A lot.

... and some more sad IT stories

Failure in the system is difficult to predict, which is logical. There are failures even in the most reliable systems where the powerful infrastructure is involved. But here the frequency of failures can be reduced, and significantly, using redundancy systems. In a reliable infrastructure, any part of it can (and should, in theory) be redundant, including power, cooling, etc. High-quality data centers use N + 1 and N + 2 schemes to ensure a high reliability system. Requirements for the reliability of the infrastructure of data centers are growing as the price of forced downtime is growing. However, problems still happen.

For example, in the same 2013, one of the largest hosting providers in the world stopped working. The data center of the company, located in Utah, USA, had problems as a result of a hardware failure during maintenance work on the server. And this led to a number of equipment outages throughout the data center. As a result, a huge number of web services and sites at the time ceased to function. This failure cost the hosting provider a considerable amount.

And immediately after the Xbox One gaming console entered the market, the load on the company's servers increased (and very significantly). Because of this, the Windows Azure cloud service also began to fail. For a whole day, there were problems with the work of Xbox Live - sometimes data was not saved, sometimes it was not loaded, multiplayer in games did not work.



In 2015, the well-known company Vtech, which produces toys and electronic devices for children, suffered. Then someone hacked into the company's server, and 4.8 million records from the customer database were stolen. In addition, data on 200,000 children were abducted (their names were indicated by parents during registration).

As it turned out, the company did not follow the principles of information security very carefully. Weak passwords, weak encryption, few backups - all this led to big problems. Vtech to some extent lost the trust of customers and investors, disgruntled by the negligence of employees.

Already this year another case of irresponsible work became known. The leader of a small hosting company, which served about 1,500 clients, was guilty of data loss. Marco Marsala (Marco Marsala) in one of the evenings loaded with work launched the command rm -rf {foo} / {bar} on all servers, and the variables {foo} / {bar} were not set (by mistake). As a result, all data was deleted from all servers.

By an unfortunate coincidence, backup drives were mounted to the servers. All this data has also been erased. The affected owner of the company asked other users if the rm -rf {foo} / {bar} command could be used to restore the information. It is clear that nothing good other users did not tell him. As a result, inattention has led to the fact that the business had to be closed (this was reported by the same Marsala). So there is little to do backups, they still need to be stored in a safe place. In the comments, users indicated that “If you have only one backup, you do not have a backup,” which, in fact, is true in most cases.

And not only small companies have problems. For example, one of the world's largest banks, Barclays, was fined several million US dollars a couple of years ago. The reason is a partial loss of business correspondence for 10 (!) Years. The company lost data due to imperfect data storage. Technical failure - and letters are lost.

And it would be okay if only banks lost letters - after all, financial companies, although they should have a perfect telecommunications infrastructure, are not the creators of email services, like Google, for example. Yes, yes, this company is also far from ideal in terms of storing user information. In 2011, Gmail, Google's email service, made thousands of its users worry.

Some of them, logging into your account, did not see any letters or contacts . Then it was affected, according to Google, "only 0.08%" of the total number of service users. But at that time, 193 million people used Gmail, and even hundredths of a percent of this number is the population of a small city. One of the clients of the service then complained about the loss of 17,000 letters - it was all his activity for all time.

Most of the company's data was returned, because Google has put data backup at the forefront of the company's work. But some users still remained with problematic accounts, plus Gmail’s reputation suffered.

Causes of data loss and backups in companies

Among the most common causes that lead to the loss of data and backup copies of data, we highlight five main points. These are carrier failure, human error, software error, hardware error and problems with network equipment.

Carrier failure

This is one of the most common causes of data loss. Most often, this problem occurs when the carriers are not monitored, and the condition of such equipment is not regularly checked. Failure to comply with the manufacturer's instructions, careless handling of the media - all this can cause a failure.

Human factor

This is the second most common reason for the failure of data backup systems, as a result of which information either does not back up at all or the backups themselves are erased. In order to avoid such problems, the staff must be qualified, we need clear instructions and plans to be carried out.

Software failure

This is also one of the most common problems. Updating software, replacing one program with another, adding a software module to the backup system - many reasons for failure. And the more complex such a system, the higher the probability of failure.

Hardware failure

Similarly to the previous problem, the hardware may refuse to replace a hardware module, due to the physical obsolescence of the hardware systems, or for no apparent reason.

Network failure

Incorrectly formed script, failure of network equipment, some problems with protocol compatibility or dozens of other reasons may affect the backup data progress.

Individuals are no better than companies (and maybe even worse) with respect to creating backup copies of data. Back in 2015, Backblaze conducted a user survey, during which it was discovered that even 39% of users completely copy their data even every year. Every day, it makes only 8% of respondents.

As output

The company's own telecommunications infrastructure is difficult, expensive and time consuming. Of course, there are times when you simply cannot do without your data center. But in the overwhelming majority of cases, it is easier for companies to use an existing infrastructure, including backup systems, than to deploy their own system.

In any case, for whatever purposes you need significant computational power, we have the “ Virtual Data Center ” service. Infrastructure as a service ( IaaS ) is not a new direction, but we benefit from a holistic approach, ranging from specifically IT problems, such as transferring corporate resources to a “Virtual Data Center”, to legal issues, such as consulting on current RF legislation in the field of data protection ( VDC.152 ).

One of the positive aspects is the absence of the need for a company using IaaS, the need to develop and expand its own telecommunications structure. All this makes the service operator. And yes, the backup storage depth (and they are created every day) is 14 days. In most cases, this is sufficient to recover any data lost during any failures. You can not make a backup - we will do it, the company Safedata. Just in case, we note that backup is a paid service, it needs to be connected.

Well, successful backups to all, comrades readers.