Amazon let down diesel generators

Amazon has explained the reasons for the blackout of the data center in the East-1 region of Northern Virginia, which resulted in Instagram, Netflix, Pinterest and other sites being offline on June 30.

The reason for the incorrect operation of diesel generators, more precisely, the distribution board (switchboard), which was supposed to switch the system from one channel to another. After the power surge, “the generators started up normally,” but if one of the data centers East-1 was powered from the generators without incidents, then in the second data center, “each generator individually could not provide a stable voltage after they were started,” Amazon explains. As a result, the network did not automatically switch to power from the generators, and the UPS soon ran out of charge.

Diesel generators were installed in the data center at the end of 2010 and the beginning of 2011, before installation they were tested by the manufacturer. After installing them in the data center, they were again driven through the eight-hour tests, everything worked fine. On May 12, these diesel generators were tested again in the field, when the basic power supply of the data center was completely turned off, everything was fine too.
')
Now the generators will be repaired and re-certified by the manufacturer, or they will be replaced by others. Amazon promises to make every effort to ensure that this situation does not happen in the future: the window of time will be increased, which is given to diesel generators to gain power to achieve a stable voltage before automatic systems decide whether to switch to generator power or not. In addition, the engineering staff will be increased in the data centers so that if there is a power outage, they can, if necessary, manually start the generator and switch the system to power from it, so that UPSs will not be able to discharge and the data center clients will not suffer.

via Wired