About the data center cleanly: how we solved the problem of dust in the server rooms of the data center

Hi, Habr! I am Taras Chirkov, director of the data center Linxdatacenter in St. Petersburg. And today in our blog I will talk about the role that the maintenance of cleanliness in the room plays in the normal operation of a modern data center, how to measure it, achieve and maintain it at the right level.

Purity trigger

Once a client of a data center in St. Petersburg approached us about a layer of dust at the bottom of the rack with equipment. This was the starting point of the investigation, the first hypotheses of which suggested the following:

• dust enters the server rooms from the soles of the shoes of employees and customers of the data center,
  
• entered through the ventilation system
  
• both.
  

Blue boot covers - to the dustbin of history

Started with shoes. At that time, the problem of cleanliness was solved in the traditional way: a container with shoe covers at the entrance. The effectiveness of the approach did not reach the desired level: it was difficult to control their use by the data center guests, and the format itself was inconvenient. Quickly abandoned in favor of more advanced technologies in the form of a quilting machine. The first model of such a device we installed failed: the car too often tore the boot covers when trying to put them on shoes, its use was more annoying than it made life easier.
')
Appeal to the experience of colleagues in Warsaw and Moscow did not solve the problem, and as a result, the choice was made in favor of the technology of fusing thermofilms on shoes. With the help of a thermofilm, you can wear “shoe covers” on shoes with any sole - up to the thin female heel. Yes, the film, too, happens to slide off, but far less often than the classic blue shoe covers, and the technology itself is much more convenient for the visitor and more modern. Another important (for me) plus is that the film easily covers the largest shoe sizes, unlike traditional shoe covers, which are torn when trying to put them on the 45th size. For greater modernity of the process, bins were installed with automated opening of the lid on the motion sensor.

This process looks like this:


Guests immediately appreciated the innovation.

Dust in the wind

Having put in order the most obvious channel of possible pollution of space, we took up more subtle matter — air. It is likely that a significant part of the dust enters the server through ventilation due to insufficient filtration, or is brought from the street. Or is it all about low quality cleaning? The investigation continued.

We decided to take measurements of the content of particles in the air inside the data center and invited a laboratory specializing in air quality monitoring in clean rooms for special purposes to carry out these works.

Laboratory staff measured the number of control points (20), made a schedule of sampling to track the dynamics and create the most accurate picture. The cost of the full measurement process by the laboratory was about 1 million rubles, which seemed to us completely inexpedient, however, it gave a number of ideas for independent implementation. Along the way, it became clear that the laboratory is good, but the tests must be carried out in dynamics and constantly apply to their services is extremely inconvenient.

Looking at the planned actions of the laboratory, we decided to look at more utilitarian devices for independent work. As a result, we managed to find the necessary tool for this task - an air quality analyzer. Here is this:


The device shows the content of particles of different diameters (in micrometers).

Revising Standards

This device analyzes the number of particles, temperature, humidity and shows the results in units of measurement, respectively, with the ISO standards for this parameter. The display shows the levels of particles with different diameters in the air sample.

In parallel, we sinned on filters: at that time, G4 filter models were used inside the server rooms. This model provides a rough air purification, therefore, allowed the possibility of missing particles, leading to pollution. We decided to purchase for the sample the F5 fine filters used in air conditioning and ventilation systems as filters of the second stage of purification (additional purification).

The investigation was conducted - you can start the control measurements. As a guide, we decided to use the requirements of ISO 14644-1 standard for the number of suspended particles.


Classification of clean rooms by the number of suspended particles.

It would seem - measure and compare the table. But not everything is so simple: in practice, it turned out to be quite difficult to find standards for clean air for the server rooms of the data center. In an explicit form, it is not spelled out anywhere, by any organization or sectoral institute. And only on the Uptime Inside Track internal forum (people who have been trained under the Uptime Institute programs have access to it) there was a separate discussion on this topic. According to the results of its study, people tended to focus on the ISO 8 standard - the last but one in classification.

The first measurements showed that we underestimated ourselves - the results of internal air inspections showed compliance with the requirements of ISO 5 in internal premises, which significantly exceeded the standards desired by participants of the Uptime Inside Track. At the same time - with a large margin. We have a data center, not a biological laboratory, of course, but in order for the concentration of particles in the air to be equal to ISO 8, this should be an object of a minimum “cement plant” class. And how to apply the same standard to the data center is not very clear. At the same time, we obtained the result at ISO 5 by taking measurements at air filtration with G4 filters. That is, the dust cannot get into the racks through the air, the F5 filters turned out to be redundant, and they were not even used.

A negative result is also a result: we continued to search for the cause of pollution in other directions, and included air quality control in quarterly inspections, combining with checks of BMS sensors with verified devices (ISO 9000 requirements and customer audits).

Below is an example of a report that is filled out based on the data obtained during the measurement. For greater accuracy, measurements are made by two instruments - Testo 610 and a BMS sensor. Limit values ​​for instruments are indicated in the table heading. The deviation of these parameters is automatically highlighted in color to facilitate the identification of problem areas or time periods.

Everything is clear here: the difference in instrument performance is minimal, and the concentration of particles is much lower than the marginal limit.

Via the back entrance

Since there were other entrances to the clean rooms in addition to the main entrance for customers, where we installed a quilting machine, there was a need to prevent dirt from entering the data center through them.

It is inconvenient to put on / remove shoe covers during procedures for unloading equipment, so we found an automatic machine for cleaning soles. Conveniently, functionally, but the human factor has an effect in the form of the non-binding approach to this apparatus. In fact, the same as with shoe covers at the main entrance.



To solve the problem, we began to look for cleaning options that could not be avoided: sticky carpets with detachable layers coped with this best of all. In the process of authorization at the entrance door, the visitor necessarily stands on such a rug, removing excess dust from the soles of his shoes.


The cleaners tear off the top layer of such a rug every day, there are 60 layers in it - enough for about 2 months.

Visiting the Ericsson data center in Stockholm, among other things, I paid attention to how they solve these issues there: along with the detached layers in Sweden, they use reusable Dycem anti-bacterial carpets. The idea was liked because of the principle of reusability and the ability to provide a large coverage area.


Magic antibacterial carpet. It’s a pity, not a plane, but it could have been - at such a price!

With difficulty, we found representatives of the company in Russia and estimated the cost of the solution for our data center. As a result, we received a figure of almost 100 times more expensive than solutions with multi-layered carpets - approximately the same 1 million rubles as in the project with air purity measurements. In addition, it became clear the need to use special tools for cleaning, of course, available only from this manufacturer. The solution also fell away on its own, settled on a multi-layered version.

Manual labor

I would especially like to note that all these measures did not abolish the use of labor of cleaners. As part of preparing for the certification of the Linxdatacenter data center according to the Uptime Institute Management & Operations standard, it was necessary to clearly regulate the actions of the cleaning service staff in the data center. Detailed instructions were made to prescribe where, what, and how to do it.

A couple of excerpts from the instructions:





As you can see, everything is written down, literally every aspect of the work in a particular room, cleaning products, materials, etc. that are permissible for use. Not a single detail remains without attention, even the smallest. Instructing - under the signature of each employee of the service. In server rooms, electrical rooms, etc. they are removed only in the presence of authorized employees of the data center, for example, the engineer on duty.

But that's not all.

Also in the list of measures to ensure cleanliness in the data center: rounds with a visual inspection of the premises, including weekly racks for the detection of scraps of wire left inside them, remnants of packages from equipment and components. For each such episode, an incident is started, the client receives a notification about the need to eliminate violations in the shortest possible time.

Also, we have created a separate room for unpacking and setting up equipment - this is also part of the company's cleaning policy.

Another measure that we learned from the practice of Ericsson is to maintain constant air suppression in the server rooms: there is more pressure inside the premises than outside, so that there is no traction inside - we will tell about this decision in more detail in a separate material.

Finally, we have got robots-assistants for premises that are excluded from the list of cleaning staff available for visiting.


The grille on top not only gives +10 to the protection of the robot, but also allows it not to get stuck under the vertical cable trays of the racks.

Unexpected find as output

Cleanliness in the data center is important for the operation of server and network equipment that pulls air through it. Exceeding the permissible levels of dusting will lead to the accumulation of dust on the components and the total temperature increase of up to 1 degree Celsius. Dust reduces cooling efficiency, which in terms of a year can turn into noticeable indirect costs, as well as affect the resiliency of the facility as a whole.

It can be said that this is a speculative assumption, but the experts of the Uptime Institute, who have certified the Linxdatacenter data center according to the quality standard of operational management (Management & Operations), pay very close attention to cleanliness. And it was even more pleasant to receive the highest praise in this area: our data center in St. Petersburg seriously exceeds the certification requirements. An expert from the institute called us “the cleanest data center that he saw,” moreover, our data center uses Uptime as an example of solving the issue of server room cleanliness. Also, we easily pass any client audit on this parameter - the most serious requirements of the most capricious clients are met with excess.

Let's go back to the beginning of the story. Where did the pollution come from at the very beginning of the article? The part of the client’s rack that caused the launch of the entire project “cleanliness in the data center” has been polluted since the moment it was imported and installed into the data center. The client did not clear the rack by the time of entry into the server room - when checking the neighboring racks installed at the same time, it turned out that there is the same situation with dust. This situation was the reason for adding a cleaning control point to the customer checklist for the rack installation. The likelihood of such things also should never be forgotten = forewarned, therefore, armed. It's all about “cleanliness and dictatorship” in our data center, in the next article I’ll talk about pressure sensors, for now ask questions in the comments.