Geekly Articles each Day
📜 ⬆️ ⬇️

Protection against fires in data centers, or how data integrity is ensured



After a sensational fire at the Dataline data center in Borovoy on June 5, we received a flurry of calls from concerned customers. All were worried about one question: "Is it worth preparing for the worst?" Despite the fact that Cloud4Y has nothing to do with the OST data center, we decided to tell about the features of such incidents. What really can threaten customer data and how cloud providers solve the issues of ensuring uninterrupted operation of servers.

A data center (data processing center), or data center, is a building, structure, or room in which servers and network equipment are installed, and which is designed to process, store, and distribute information. The data center is designed for continuous operation under high load, and therefore must be protected from any emergency. If he “flies”, economic and reputational losses for a company or companies that used its services may be critical.
')
Any simple, even a second, can lead to a failure of the services it serves, related not only to business processes, but also to the safety of people. Therefore, there are two main requirements for data centers: resiliency and cost of operation.

Properly designed fire safety system is an important component of fault tolerance, so data centers spend a lot of money on its improvement. And it helps - fires in data centers happen very rarely, and user data suffers and does in exceptional cases.

Why burn data centers


The main causes of fire:


You need to understand that a good modern data center is a large area, which is penetrated by kilometers of cables and is filled with electronic equipment (cabinets, racks) with a high concentration of devices and power consumption. All this equipment is hot during operation, and therefore is a potential source of ignition. Do not forget about the UPS or DDIBP with fuel, as well as external causes of fire in the data center: lightning, flooding, the human factor.

However, practice shows that there are practically no fires directly in the protected premises of the data center, the level of protection is so high there. In the data centers install sensitive fire detectors, combined into a common warning system about fire and fire suppression. There is also an obligatory regulation, according to which a complex of organizational measures is carried out, including a constant visual inspection of equipment, verification of compliance with fire regulations and rules for the operation of electrical installations.

And Christoph Kainz, an independent expert on fire risks from Germany, believes that in about 80% of cases the cause of fires in data centers lies outdoors with IT equipment. His experience shows that fires occur due to poor separation of engine rooms from neighboring rooms, a weak fire system in these rooms and a high fire load in them.

Obviously, data centers seriously invest in fire safety, so fires directly on racks are extremely rare. Therefore, user data usually remains intact.

Fire scenarios in the data center




There are three scenarios for the development of fire in data centers. For each of them, strictly defined detection tools are applied that are effective precisely for this situation.

Scenario number 1: smoldering (wires or circuits) . Smoke is present in the air in a minimum concentration. A fire at this stage will be able to detect the aspiration system for early detection of fire. Since damage to electronics often causes not a flame, but a gas released during smoldering and burning, which oxidizes electronic contacts.

Solution : Modern data centers use gas analyzers built into the early fire detection system.

Scenario number 2: strong smoke rack space . Smoke detects the conventional smoke detectors installed inside the racks. The signals they send are processed by the server equipment monitoring program installed in the data center.

Solution : Sensors are used as a source of additional information to monitor all systems of this object. Sometimes - to activate a stand fire extinguishing system.

Scenario number 3: a strong smoke room data center . Detected through addressable or untargeted sensors of a common fire alarm system installed inside the premises of the data center. Fire panels, which act as a central receiving and controlling equipment, when fire is detected, automatically start active fire extinguishing systems.

Solution : The extinguishing process already covers the entire data center space. Depending on the characteristics of the premises, different fire extinguishing agents are used.

We emphasize once again that the combustion products of cable insulation can cause significant damage to electronic equipment. In case of fire, out of 1 kg of PVC several hundred liters of flue gases are released, including hydrogen chloride HCl. The latter, reacting with the surrounding atmosphere, is synthesized into hydrochloric acid, which causes corrosion of electrical contacts and even short circuits of the latter. Needless to say that such processes should be detected and eliminated as early as possible?

Therefore, at the NPP, for example, non-combustible cable products ( STO SRO-S 60542960 00030-2014 , STO 1.1.1.01.001.0902-2013 ) are used for the installation of cable lines in the installation sites of microprocessor technology, computers, and others. polymer compositions that do not emit corrosive gases during smoldering. In the data centers relevant GOST 31565-2012 “Cable products. Fire Safety Requirements ”, which determines the type of cable products that do not emit corrosive gaseous products during combustion and smoldering.

Sprinkler and drencher fire extinguishing systems on racks cannot be used, because if they are triggered, expensive equipment will be flooded with water and irreparably damaged. But then how to extinguish?

How to extinguish data centers




Automatic fire extinguishing systems begin to work in the event that the development of a fire cannot be stopped by other means, for example, by turning off a puffed-up server or air-conditioning system. As we have already said, an improperly selected fire extinguishing installation, when triggered, is capable of causing damage comparable and sometimes even exceeding damage from the fire itself. Therefore, you need to choose the installation, taking into account the following factors:


At the moment there is no such means of extinguishing a server that would meet all these requirements. Therefore, when equipping data centers, one or several options are selected from among the five main methods of eliminating fires:

Hypoxic method


The essence of this method is to fight against ignition by creating and maintaining an atmosphere in which a fire cannot occur. Nitrogen is introduced into the room, which reduces the oxygen content to below 14%. Nitrogen is constantly produced from atmospheric air by a special generator. In such an atmosphere, fire cannot arise and spread, and at the same time such a level of oxygen is sufficient to work in the server room.

The hypoxic method wins over other fire extinguishing methods in all parameters, except for one - cost. Not every company is able to spend such an amount of money to fight a fire.

Insulation


An effective method that works on the principle of "replacement of oxygen." In a room engulfed in fire, pure inert gas or a mixture for fire extinguishing is supplied. The oxygen level falls below 14% and the flame goes out. Nitrogen, argon, argonite or inergen is used. Of the minuses again can be noted the possibility of chemical reactions dangerous to humans.

Inhibition


The method, which involves the binding of active sites, which leads to the termination of a chain reaction of combustion. If it is simpler, then a halogenated gas is injected into the room, which inhibits chemical reactions in the flame, suppressing the active centers (radicals and atomic particles that have free valences), thereby interfering with the combustion process. Freons (fluorinated hydrocarbons) are commonly used as inhibitors. Recently, the substance Novec 1230 (known as “dry water”) has also become popular.

Note that during the chemical reaction of inhibition, the release of by-products is possible, which can be dangerous for people, and also leave a raid on the protected equipment.

Cooling


One of the youngest ways to extinguish server rooms, widespread in Europe, in data centers such as TCN Eemsdelta and Telecity IV in the Netherlands, the CX2 Cyberjaya science and technology test site in Malaysia and others. During a fire, water mist is sprayed onto the burning area (streams of fine mist water), which leads to a decrease in the level of oxygen at the local level and cools the zone of origin of fire. Distilled water is used as water, which is allowed to extinguish electrical equipment with a capacity of up to 10 kV.

This method uses 90% less water than sprinkler systems, eliminates the pipeline flow in the daily mode and cheaply when refilling the system. But also need to mention the high cost of the system. It involves the use of one to several pumping stations that maintain a constant pressure in the pipeline, which also has serious technical requirements. Partly because of this, this method is not widespread among Russian data centers. We maintain a persistent distrust of water as an extinguishing agent for electrical equipment. Even fine water can condense into drops, so it will somehow be damp in the place of fire, which is unacceptable in the server room.

Powder / Spray


The least common method of extinguishing in data centers. When ignited, the powder chemistry is released and the aerosol is sprayed. Both powder and aerosol on the surface of burning hot objects form a film that prevents the penetration of oxygen, which reduces the likelihood of re-ignition. In the server it will become a problem, since the powders and aerosols penetrate into any equipment and settle on the internal components that are not protected from contact with aggressive substances. After such extinguishing, the equipment will gradually fail due to corrosion and the occurrence of short circuits in electrical circuits.

By the way, in the transformer and DDIBP systems powder fire extinguishing systems are usually installed, and in the office premises, corridors and common areas - traditional sprinkler systems.

What is the most effective fire solution?


In general, the obvious leader now is the method of gas extinguishing . Gas does not harm electrical equipment and works great even in remote premises. Also it is necessary to take into account the fact that the server rooms in the data center operate without the constant presence of personnel in them, and the fire suppression is carried out with the equipment running (energized). Due to the compactness of gas installations, they can be scaled for a specific object of protection and a supported temperature range of work from -40 to +55 ° C, protecting modular and container (mobile) data centers.

Gas extinguishing systems can be organized according to two principles:


In most cases, modern gases for extinguishing fires use the gases listed below:


The choice of the “right” extinguishing agent is done only after analyzing the protected object.

The most famous "fire victims"




03/27/2010
Fire in the Future Technologies data center. As a result, the servers suffered from the fire and the water that extinguished it. More than a day about 2500 thousand sites were disabled. Lay hosting providers hosting.ua, ostia.ru, onelim.net, provisov.net, imhoster.net, alekshost.ru, onlinehoster.net, xlhost.ru. Surprisingly, one of the most modern fire extinguishing systems was installed in the data center, which did not work because it was turned off manually. It very often worked falsely and the data center employees disconnected it. As a result, the fire had to deal with firefighters. The damage was estimated at tens of millions of dollars. This fire was the impetus for the creation of higher-quality detectors and served as a reason to rewrite more than one internal instruction on fire safety.

11/05/2010
During a fire at the new computer center of the ERGO insurance group in Dusseldorf, 28 people were injured on Friday morning. 20 employees were hospitalized with severe carbon monoxide poisoning. No problem reported with user data.

06.07.2012
In Seattle, Microsoft’s homeland, a fire broke out in the power system of a large data center, which caused the search query functionality in the new Microsoft Bing search engine to fail. User data is not affected again.

04/20/2014
The fire in the building of the Samsung SDS data center, which is located in Gwacheon (South Korea), caused the failure of smartphones, tablets and smart TV around the world. At the same time, Samsung.com went offline as well. Downtime data center lasted for several hours, after which most users again received full access to the functionality of the devices. Samsung.com is also back online.

02/10/2015
Attack intruders suffered a large Danish commercial company. The attackers set fire to the company's office, and only coal was left of most of the building. The server farm was surrounded by fire, which in 60 minutes destroyed everything outside of it - including power and telecommunication cables in the external room. The outer walls of the server room were so hot at that moment that the fire brigade decided to drill a hole in the door to see if there was a fire inside. When the room was eventually opened, all the IT equipment turned out to be safe and sound. Extremely high temperature during a fire activated the fire extinguishing system of the modular data center, which flooded the room with inergen (a mixture of nitrogen, argon and carbon dioxide). The pressure equalization valve allowed the excess gas to go outside to eliminate excess pressure. At the same time a small cloud of smoke got into the room, due to which several dark spots appeared on the walls. After the deployment of new power infrastructure and network cables, the equipment was restarted. In the end, it took about three days to restart the server farm.

06/30/2015
A strong fire in a room with telecommunications equipment inside the data center of one of the largest British operators BT Group brought the entire data center in Belfast (United Kingdom) out of service and left many of the company's clients, including government agencies, without access to communications and the Internet. Tibus (service provider) and Translink (engaged in public transport), Belfast City Council, Northern Ireland Electricity (energy company) were particularly hard hit. Because of what there was a fire, the company did not report.

11/24/2015
The fire in the Delta Telecom data center in the capital Baku deprived Azerbaijani users of Internet access. Downtime lasted for eight hours and affected 78 percent of Azerbaijani networks. We are talking about more than 6 hundred networks that used one key connection between Delta Telecom and Telecom Italia Sparkle. After this incident, access to Internet services was possible only through the use of channels of local mobile operators Backcell and Azerfon. Problems with the Internet have arisen due to the low number of networks that connect the country with external traffic exchange nodes. A similar situation is now typical of many neighboring countries such as Iran, Georgia, Armenia and Saudi Arabia.

08/17/2016
Due to the smoke in the data center of the Canadian state organization Shared Services Canada, which is responsible for servicing the government's IT infrastructure, several departments were unable to provide services to citizens of the North American state in normal mode. The incident triggered a massive shutdown of government websites and payroll systems. In addition, the internal e-mail server for civil servants is offline. According to the Canadian Press news agency, about 50,000 public service workers were unable to receive and send emails during the day. Downtime has been affected and the Ministry of Transport of Canada.

03/16/2016
Fire in the data center Selectel in St. Petersburg. A fire occurred during construction. The fire spread to the roof and facade of the building above the office premises. Because of the fire, the street was blocked, but customer data was not affected.

03/06/2018
In the southern state of Brazil, Rio Grande do Sul, the city of Porto Alegre, there was a fire in the BRDigital data processing center, part of the CommCorp group of companies. Firefighters who arrived at the scene evacuated people and de-energized the 13-storey building in the very center of the city. Soon the fire was localized and liquidated, and the police blocked the building for an examination scheduled for Wednesday. Companies affected by the fire and located in this building, began to resume its activities only on March 9, on Friday. One of the customers of this data center, Rafael Azeved, wrote: "The only fire system that existed in the data center was a fire extinguisher from the kitchen." If interested, here is the story of the last minutes of one of the servers:

[15:22 pm] - IPMI: 0
[15:23 pm] - IPMI: 1
[15:24 pm] - BCM
[15:25 pm] - IPMI: 0
[15:25 pm] - IPMI: 1
[15:25 pm] - IPMI: 2
[15:25 pm] - IPMI: 3
[15:25 pm] - IPMI:
[15:26 pm] - IPMI:
[15:26 pm] - IPMI: > 180
[15:26 pm] - IPMI: BCM

05.06.2019
The fire in the OST data center led to disruptions in the services of Mail.ru. To extinguish it, we had to disable most of the main mail servers. 80% of complaints to the Down Detector service were related to the unavailability of Mail.ru mail, 16% of Mail.ru website did not open at all. Among the victims was the company QIWI, whose data center was also located in the building. It took them about 30 minutes to switch all operations to the backup system and restore the regular mode for working with payments. It is alleged that user data was not affected by the fire.

And what do cloud providers do in the meantime?


Data center operators are doing everything necessary to minimize the risk of a fire. But the providers also do not sit idly by, but come up with additional ways to protect customer services.

About what measures have been taken in the company, says Cloud4Y Commercial Director Artyom Goncharenko:
“We cannot say that we are not afraid of fires. Afraid of course. This is primarily the creation of additional work on virtually all departments of the company. However, everything that concerns the performance of customer services, we take seriously. Critically serious.

In fact, the risk of loss of the data center, the risk of fire safety is calculated by us in advance. Each client has protection against data loss in the form of daily backups made automatically in a separate physical data center that is 10 km away.

There is also a life hack that works for free only in Cloud4Y. Each of our clients has the opportunity to place resources in several data centers simultaneously, i.e. the client can free of charge (no need to pay money for the opportunity provided, only resources are paid for) to protect themselves by placing resources in both data centers and setting up duplication at the service level.

This can also be done by our technical support specialists. In fact, we provide the ability to build a distributed infrastructure in your cloud, which allows you to protect user data in case of an emergency in one of the data center.

By the way, daily backups will be formed and stored in a cross-system with data center1 in data center2 and vice versa, backups from active data center2 systems will be sent to storage in data center1.

The ability to place their facilities in two independent data centers at once is a valuable, but extremely rare service. Therefore, if you are hosted not in Cloud4Y, then I recommend thinking about additional placement of a backup copy of the infrastructure in other data centers from another provider. Whether it is a company with on-premise equipment, or a cloud business, moving the backup to a separate physical room will guarantee data security in case of accidents and incidents. Simply placing a backup on a separate physical server or even on another floor of a common building is not enough. If the emergency will affect the entire building, it will not matter where the backup was stored. The data will disappear.

Many companies come to the practice of infrastructure duplication. We are approached by large customers who want to place their backup away from the main infrastructure. Here we offer the following options to choose from: the first and simplest is backup storage, the second is BaaS (backup-as-a-service) with the guarantee of providing the required amount of capacity, the third is the “backup data center” service with strictly specified in the contract requirements for recovery time and the maximum number of lost minutes of the company.

Do you ask for advice? It is very simple: store your files and their backups in different places, hosting providers, data centers, countries. It will save your nerves, time and, most importantly, business. ”

Conclusion


Cloud4Y for data storage customers uses a network of data centers in Russia and Europe, certified at the level of reliability Tier 3. How to ensure fire safety in data centers Tier 3:


The fault tolerance level of the data center Tier 3 reliability level is 99.982%. This is achieved not only by duplicating the systems, but also by a well-thought-out fire safety concept.

Source: https://habr.com/ru/post/456414/

More articles:


All Articles