10 most unexpected interruptions in data center operation
One day, one way or another, the electrical network fails along with the backup systems, or the engineer makes a mistake, and the data center “crashes.” And although technology in the industry is constantly evolving, reducing the frequency of problems, sometimes something suddenly goes wrong. But there are cases when terrible failures are caused by completely unexpected and surprising reasons.
And here is a list of the 10 most incredible reasons for such failures:
')
The extra second is sometimes added to the World time settings to take into account the change in the speed of rotation of the Earth. Adding one second to the course of the world's atomic clocks in 2012 brought a lot of trouble. Then there were interruptions in the work of popular web resources, including LinkedIn, Reddit, Mozilla and The Pirate Bay. In Australia, 400 flights were delayed on average by two hours due to the fact that employees had to switch to manual check-in of passengers.
In fact, the squirrels that cause malfunctions in the work of the data center are not to be said to be quite rare. They gnaw everything, including the wires that are used to transfer data. In 2010, during a report at the O'Reilly Velocity conference, Mike Christian, who runs Yahoo's business continuity service, said: “The fried protein smashed half of our data center in Santa Clara.”
Shipment of servers can be quite troublesome. NaviSite, now owned by Time Warner, acquired hosting provider Alabanza in 2007 and had to move servers from their main data center in Baltimore to its site in Andover, Massachusetts.
In fact, they disconnected the servers, loaded them into trailers and transported the servers more than 420 miles away. Many sites that hosted at Alabanza, did not respond to requests, while the physical movement and commissioning in a new place lasted.
Another problem associated with the move arose several months earlier when Hostway transported ValueWeb servers from Miami to Tampa. Hostway after noted that more than 500 servers had hardware problems after installation on the new site.
Between the continents, the signal is transmitted by powerful underwater cables . These cables are quite durable, given their location. However, there was one case where the ship just dropped anchor on one of the cables. In 2008, there was a whole epidemic of damaged submarine cables, which led to interruptions not only in the work of the data center, but also left entire regions without communication.
Nianet, a Danish Internet service provider, disconnected when thieves punched a hole in the data center wall in Toastrup to get inside and steal something. According to media reports, they left, capturing a bunch of network cards. How thieves were able to break through the wall and why they took only network cards, still remains a mystery.
There was at least one case where the work of the data center was disrupted due to smoldering mulch. Perth iX data center in Western Australia stopped working for an hour after the VESDA fire alarm system triggered (Very Early Smoke Detection Apparatus, a very early smoke detection device). It turned out that the system caught the smoke of smoldering mulch (various biomaterials like foliage, compost, etc.) from the garden bed adjacent to the outer wall of the object, which caught fire, most likely due to an unstuffed cigarette butt.
In 2007, Rackspace, a company known for its phenomenal trouble-free operation, experienced several hours of inactivity due to a truck driving into one of the power transformers, followed by an explosion.
The backup power system attempted to maintain operation, but two chillers could not be started. Several large sites did not work on the network at the same time.
In 2009, one negligence of a little-known Czech Internet provider in the BGP settings caused a number of small interruptions in the work of several large hosting companies. The Czech provider Supronet "for several hours single-handedly arranged a global crisis on the Internet," said Renesys.
In 2007, two masked men broke into the data center in Chicago and stole a bunch of computer equipment. This data center belonged to the old hosting company CI Host, which currently does not exist independently.
An employee who worked alone during the night of the robbery was reported to have been hit by a stun gun, while others say he was beaten. More than 20 servers were rendered, which effectively destroyed many sites.
After the robbery, there were a lot of rumors about what happened. It was rumored that the robbers got inside with a saw (which will happen to Nianet a few years later). CI Host, on the other hand, said that two robbers hid in a mechanical cabinet (elevator). It was believed that the company itself arranged this crime with the aim of committing fraud, but these rumors were not confirmed.
The most amazing thing about this whole thing was that it was not the first time.
“People are talking about four robberies, which is wrong!” The company’s general director told the WHIR magazine about this. “A robbery is when property is taken away by force or by threats. Technically, CI Host was robbed just twice in two years. Twice there were hacks, things were stolen, but they were not robberies. ”
It's hard to talk about Hurricane Sandy. We hope that such an extraordinary phenomenon will no longer cause chaos in New York. But after the data center's disruption caused by this storm, we heard several amazing stories, such as the Brigade story with buckets (Peer 1 Hosting employees carried buckets of diesel fuel to a height of 17 floors to launch an emergency data center generator of the company).
Usually, reaching so far to the north, the storms almost disappear. However, Sandy moved to the north, taking on some of the features of tropical storms and gaining more and more strength. A strong current sent a storm to the coast, it was a full moon, the tide was stronger than usual, the waves were just huge.
This type of storm has never been observed in the Northeast. This is even more surprising than a smoldering mulch, a truck that has strayed off a highway, roasted in power cables. Some things in the world we can never foresee.
And here is a list of the 10 most incredible reasons for such failures:
')
Error per second
The extra second is sometimes added to the World time settings to take into account the change in the speed of rotation of the Earth. Adding one second to the course of the world's atomic clocks in 2012 brought a lot of trouble. Then there were interruptions in the work of popular web resources, including LinkedIn, Reddit, Mozilla and The Pirate Bay. In Australia, 400 flights were delayed on average by two hours due to the fact that employees had to switch to manual check-in of passengers.
Squirrel yahoo
In fact, the squirrels that cause malfunctions in the work of the data center are not to be said to be quite rare. They gnaw everything, including the wires that are used to transfer data. In 2010, during a report at the O'Reilly Velocity conference, Mike Christian, who runs Yahoo's business continuity service, said: “The fried protein smashed half of our data center in Santa Clara.”
Moving
Shipment of servers can be quite troublesome. NaviSite, now owned by Time Warner, acquired hosting provider Alabanza in 2007 and had to move servers from their main data center in Baltimore to its site in Andover, Massachusetts.
In fact, they disconnected the servers, loaded them into trailers and transported the servers more than 420 miles away. Many sites that hosted at Alabanza, did not respond to requests, while the physical movement and commissioning in a new place lasted.
Another problem associated with the move arose several months earlier when Hostway transported ValueWeb servers from Miami to Tampa. Hostway after noted that more than 500 servers had hardware problems after installation on the new site.
Anchor against the Internet
Between the continents, the signal is transmitted by powerful underwater cables . These cables are quite durable, given their location. However, there was one case where the ship just dropped anchor on one of the cables. In 2008, there was a whole epidemic of damaged submarine cables, which led to interruptions not only in the work of the data center, but also left entire regions without communication.
"Every wall is a door." - Ralph Emerson
Nianet, a Danish Internet service provider, disconnected when thieves punched a hole in the data center wall in Toastrup to get inside and steal something. According to media reports, they left, capturing a bunch of network cards. How thieves were able to break through the wall and why they took only network cards, still remains a mystery.
Be careful when throwing bulls.
There was at least one case where the work of the data center was disrupted due to smoldering mulch. Perth iX data center in Western Australia stopped working for an hour after the VESDA fire alarm system triggered (Very Early Smoke Detection Apparatus, a very early smoke detection device). It turned out that the system caught the smoke of smoldering mulch (various biomaterials like foliage, compost, etc.) from the garden bed adjacent to the outer wall of the object, which caught fire, most likely due to an unstuffed cigarette butt.
We continue to drive
In 2007, Rackspace, a company known for its phenomenal trouble-free operation, experienced several hours of inactivity due to a truck driving into one of the power transformers, followed by an explosion.
The backup power system attempted to maintain operation, but two chillers could not be started. Several large sites did not work on the network at the same time.
Czech and mat, Internet
In 2009, one negligence of a little-known Czech Internet provider in the BGP settings caused a number of small interruptions in the work of several large hosting companies. The Czech provider Supronet "for several hours single-handedly arranged a global crisis on the Internet," said Renesys.
Pawnshop servers
In 2007, two masked men broke into the data center in Chicago and stole a bunch of computer equipment. This data center belonged to the old hosting company CI Host, which currently does not exist independently.
An employee who worked alone during the night of the robbery was reported to have been hit by a stun gun, while others say he was beaten. More than 20 servers were rendered, which effectively destroyed many sites.
After the robbery, there were a lot of rumors about what happened. It was rumored that the robbers got inside with a saw (which will happen to Nianet a few years later). CI Host, on the other hand, said that two robbers hid in a mechanical cabinet (elevator). It was believed that the company itself arranged this crime with the aim of committing fraud, but these rumors were not confirmed.
The most amazing thing about this whole thing was that it was not the first time.
“People are talking about four robberies, which is wrong!” The company’s general director told the WHIR magazine about this. “A robbery is when property is taken away by force or by threats. Technically, CI Host was robbed just twice in two years. Twice there were hacks, things were stolen, but they were not robberies. ”
Superragan Sandy
It's hard to talk about Hurricane Sandy. We hope that such an extraordinary phenomenon will no longer cause chaos in New York. But after the data center's disruption caused by this storm, we heard several amazing stories, such as the Brigade story with buckets (Peer 1 Hosting employees carried buckets of diesel fuel to a height of 17 floors to launch an emergency data center generator of the company).
Usually, reaching so far to the north, the storms almost disappear. However, Sandy moved to the north, taking on some of the features of tropical storms and gaining more and more strength. A strong current sent a storm to the coast, it was a full moon, the tide was stronger than usual, the waves were just huge.
This type of storm has never been observed in the Northeast. This is even more surprising than a smoldering mulch, a truck that has strayed off a highway, roasted in power cables. Some things in the world we can never foresee.
Source: https://habr.com/ru/post/230867/
All Articles