Video surveillance in data centers: combining the paranoia of protection with the paranoia of engineers

Engineer during regular inspection

In retail networks, now one of the main tasks of video surveillance is to detect the faces of intruders from a well-known base at the entrance.

Video surveillance in the data center went on a slightly different branch of evolution. To begin with, the guards even simply do not know what to look at, and the control, in fact, ends at the perimeter. And anything happens. I know, for example, about the case of abnormal start-up of a fire extinguishing system. We were looking for the reason for the records - it turned out that the fitter had put his elbow on the button in the corridor.
')
There are requests for someone who opened the rack. In some data center it may happen that they carry something along the aisle and touch the cable. Or here is another case: an engineer of one company served the top server in a rack, in the process a ladder fell down and struck directly on another server, as a result a connector split.

Therefore, of course, video surveillance in data centers is necessary. I'll tell you how we do it for customers, and how we organized it in our data centers.

Our experience

We started to put video control from its first data center. Initially, video surveillance should have controlled the passage of people into the control zone. The second, in fact, is the movement in the machine room and technical premises nearby. Persons at the entrances-exits.

It quickly became clear that since there are at least 30 cameras in a single data center machine building, the picture from all is almost the same. The guards cannot distinguish where that is (the inter-racks are similar), and this is incredibly exhausting. Therefore, we almost immediately began to algorithmize the main tasks.

And, of course, the guards will not understand whether the engineer is doing everything right (in his stand or not, for example, now).

They began to allocate responsible racks for themselves - if someone approaches them, the guard sees the corresponding notification. Plus, we have specificity: when the iron is right in the machine room behind the kitchen garden, the security personnel of the customer also wants to monitor the area. We still do this: we allocate elevated responsibility areas inside the data center and raise an alarm every time someone is inside or just comes to a dangerous place. One of the customers implemented it like this: the security somewhere in the central office has a screen, when an alarm goes off, it switches to the desired object, the speaker makes a sound, the SMS message drops. It is very useful to watch your counter in this way in a commercial data center.

If you have your own data center, then video surveillance is also very useful. As a rule, the entrance to the machine room is recorded in the access control system, but it can be a problem to figure out what exactly was done inside. Many random situations are needed to find the cause of problems, and sometimes even the expert himself does not notice.

On the other hand, putting a camera on each rack is also not worth it. As a rule, you don’t really need to know if the third or fourth unit was touched by the installer - it’s enough to understand that it was he who was picking at this particular rack exactly at that time. This solves the camera on the inter-rack corridor. But a couple of times we did for the blades (generally horizontal) exactly the point video control.

At first we joked that keeping records for the month is for paranoids and brakes. Then in practice it turned out that the perimeter should not be stored as long as the aisles. Two months, three months - in 99% of cases it is not necessary, but sometimes - oh, as necessary. Therefore, at the same time, we greatly increased the depth of the archive of records by machine halls, plus we began to take a “photo” when people crossed key places — these pictures with metadata are not deleted at all. There are also special customer requirements for archives, but there the protection class is almost like on defense sites.

Archives, of course, are stored with backups at geographically separate points.

In most cases, we now do not need in-depth analytics when analyzing an incident. The chief security officer comes, marks the passage, says the gap. His employee makes a selection with metadata from the access system. After 15 minutes of studying the sample, the main one says: “Mmm ... well, good." And leaves. Problem solved.

Another thing - with customers. Some need analytics. Somewhere it is important to count people, somewhere - anxiety when getting into areas of increased responsibility, and so on.

Another good example of a task is the maintainers in data centers. At one of the sites there was a problem that it was necessary to look who was doing what, but it was unpleasant to go for the engineers for 6 hours. A dry strong wind, low temperature ... - nothing to do, even in a sweater uncomfortable. They wanted to do something like an aquarium with observation posts at the entrance, the attendants would sit in a normal place 7–8 meters away from a working engineer, and would watch while sipping tea. But the customer’s security personnel said, “Let them suffer.” Not mounted.

For one of the objects we are now going to install turnstiles with 3D face recognition (biometrics), as we have at the entrance to the office building. Immediately - the two most frequent questions about the dead head (muscles sag, do not miss) and twins (differences - they have different development of facial muscles). But this is the task of the perimeter. Much more interesting is the intellectual search in the archive, the control of lines, the map of the intensity of the room. We had a part of the developments from retail networks - there, for example, by cameras it is often important to build maps of people in stores and to know which goods and where are more in demand. In data centers almost like this, only the score is not for thousands of people, but for units. But the algorithms are similar.

Unlike public places, there was no task of detecting abandoned objects, for example. Tried on tests, but only server covers and various stubs came across - not a single actual case.

On the cameras must be said separately. In the data center stable lighting, no windows. Therefore, it usually turns out quite simple and relatively cheap. Although there are customers who insisted on a fancy camera with infrared illumination. To see each connector, even if the attacker turns off the lighting. Pah-pah, so far there have been no cases.

For installation - do not put high. In the data centers are often at the top of the trays, pipe wiring and so on. Immediately thought up places lower, often - on the brackets on top, but not blocking the free opening of the doors of hardware cabinets. It is better not to install cameras in advance, after installation of non-standard servers there may be overlaps in the frame.

Banks usually have clear requirements for hardware, installation rules, their own principles of intersection of sectors, and so on. For what exactly each element is not prescribed, you just need to implement as on the plan. In other cases, we can either advise something like best practice and do as the customer’s expert decides, or we are given full observation, and we do it based on our experience with the object.

General approach

Example task: to conduct round-the-clock video surveillance in the data center.
We determine what we will observe. If this is a separate building of the data center, then we look at the perimeter, all entrances and exits to the territory. Then we look at the building itself: all entrances, exits, corridors, entrances to special zones, inter-racks, etc. If the customer wants, we can observe each rack separately. Based on what we are going to observe, cameras and methods for their installation are selected. That is, for the perimeter, these can be special outdoor cameras with good photosensitivity, wide WDR, IR illumination, etc. For indoor surveillance, where there is constant good lighting, the requirements for light sensitivity are slightly lower, but the resolution requirements increase (high resolution is required , for example, to identify a person who carried out some work in some room of the data center). To monitor a particular rack, a very high resolution camera is required. It is necessary to view each server, each button, etc.

Video Recording
We determine the archive depth, recording parameters (cad./sec., Resolution, quality, codec, alarm recording or around the clock, etc.), determine what kind of data storage needs to be provided, etc. After receiving the answers to all these points, we already decide whether it will be a separate storage (EMC, IBM, etc.), or enough internal server disks, or it will be network storage (NAS storage - Network Attached Storage), etc.

Banks keep the archive for 3 months. This comes to them from the PCI DSS standard, which regulates the security of systems that process data on bank cards (Visa, MasterCard, etc.). Clause 9.1.1 of the standard, version 3 (current, forth coming fourth).

Video analytics
With the help of different detectors, you can determine the violation of the intersection of the perimeter of the territory of the data center, you can determine the fact of passage in a special area (for example, the space in front of the customer’s racks). When a detector is triggered, alarms are generated that the security service must respond to. Also, the detectors are the cameras themselves - when an object hits a critical area, an alarm can be generated.

In general, online analytics can improve the efficiency of the security service by focusing on specific incidents or anxiety. Also, the integration of video surveillance systems with other security systems will reduce the total number of false alarms.

There are also video analytics for working with video archives. For example, using this analytics, you can quickly find movement in a given zone and at a certain time. Or determine all intersections of the perimeter for a specified time, etc.

A very simplistic schematic diagram for a small object looks like this:

Summary

Most often, customers put the following tasks:

1. Ensuring security at the facility, 100 percent control of the premises by security officers 24x7. This is done simply.
2. The possibility of debriefing in case of force majeure. For example, the customer turned off the power of the central switch, when diagnosing it turns out poor contact of the power cable. The customer wants to be sure that no one at the time of power off the rack did not open and did nothing in it. There are more serious questions.
3. Security officers often insist on the psychological factor: cameras convince visitors not to try to gain access to someone else's racks or to press something there.

If you have any questions, I will be happy to discuss them in the comments. For specific data centers you can ask immediately by mail - ANemirovskaya@croc.ru.