We design storage for video surveillance

In the previous article, we talked about the features of the organization of the IT infrastructure and the storage subsystem for video surveillance and concluded that in large projects it is advisable to use storage systems (data storage systems) for organizing the storage of the video archive.

This publication will discuss the problem of choosing storage for video surveillance, the advantages of RAIDIX-based storage and examples of their actual implementations.

The choice of storage for video surveillance

Storage Types

Storage systems can be used for a wide range of data storage tasks: file storage, video surveillance, business applications, databases, virtualization, archives and backups, etc.
')
For slow tasks with large volumes, less productive solutions with large arrays on the HDD (file and backup storage) are usually chosen. Hybrid solutions (flash + HDD) are used for virtualization and general-purpose tasks in order to meet the needs in both speed and volume. For sensitive to delays, the attachments and databases choose very fast and expensive storage - as a rule, today it is all-flash, where speed comes to the fore, the volumes there are generally small.

For video surveillance tasks, of these three categories, the primary choice is primarily the first one, which is volume oriented, in order to get a low price for a terabyte. They can also choose more expensive hybrid solutions, but there will be little confusion from them, the flash cache (or shooting range) will become clogged with streaming (sequential) storage traffic and everything will rest on slow HDDs. Fast random access all-flash storage in successive operations, of course, will also be very good, but for video surveillance that requires large amounts, this is an unreasonably expensive and inefficient solution.

Features I / O for video surveillance

In the context of data storage, video surveillance is characterized by multi-threaded sequential I / O operations:

• Each IP camera creates its own sequential data stream for recording in the video archive storage. As a result, a serious multi-threaded write load is formed.
• When viewing a video from the archive, sequential read operations consisting of one or several streams are performed.

From the point of view of the load on the storage, it is optimal when the project does not involve constant viewing of video from the archive by a multitude of operators from several workplaces. Real-time monitoring does not require reading data, and viewing from the archive takes place on a single video recording - occasionally or even permanently. In this case, the main load - 90% or more - will fall on the record, the storage system will not be exposed to a significant reading load.

Otherwise, when constant parallel viewing of multiple videos from different cameras from the archive is required (for example, a casino, a stadium, a shopping center), we will get a lot of competitive read streams and the load on the storage system will increase dramatically. Consequently, the same number of cameras will have to put a steeper store, at least twice as productive, or reduce the number of cameras per array (controller).

Problem of choice

Today on the market there are many models of storage systems from various manufacturers:

• There are vendors who deal only with data storage, and storage systems are their main or only products.
• There are IT giants who make all kinds of hardware and software, while maintaining and developing their own line of storage systems and storage solutions.
• Many large manufacturers of software and hardware for video surveillance offer their stores.

Conventionally, all storage models can be divided into two classes:

• Brand stores . In other words, storage from class “A” manufacturers or premium segment. They are cool, reliable, they have good service. They have a very cheap price tag, even on entry-level models, and serious productive models are like an airplane. The most annoying thing is that it is often impossible to install arbitrary screws (HDD) in them, they will normally stick in, they will work and only the “native” disks of the storage vendor will not be deprived of the warranty on the hardware. In fact, it will be the same server screws from manufacturers who produce hard drives, but with a proprietary sled and proprietary label of the manufacturer of the storage system, they will only cost several times more expensive. Actually, with screws for branded servers, things are the same. For projects on video surveillance, where people are accustomed to counting money and do not want to overpay at times for the brand, these solutions are not suitable. Of course, if the customer is "sitting on the pipe" - this is his case, can afford.
  
  
• Budget solutions . These are ready-made storage from “popular” storage systems vendors or prefabricated solutions on inexpensive standard (commodity) server hardware and software storage systems (SDS), paid or free. This includes products from video surveillance vendors. The prices are nice, you can put any screws, but there may be problems with reliability, performance and service. This solution is suitable for small and non-critical projects in a limited budget. For serious systems where guaranteed reliability, performance, fault tolerance and service are needed, such solutions are unacceptable, this needs to be understood.

Ideally, for a serious video surveillance project, I would like to receive a storage solution that combines only the strengths of both storage categories on the market — due quality for reasonable money. In this segment, this solution is based on RAIDIX products. Immodest, but perhaps it is a fact. Below we reveal the truth of this statement.

RAIDIX Storage Features

Architecture

RAIDIX - software storage or SDS (Software Defined Storage), which allows you to build on the basis of standard server hardware reliable, productive and fault-tolerant data storage. In principle, any x64-servers are suitable for this, including:

• 1-2 Intel Xeon processors of a suitable model and the required amount of RAM;
• one or more SAS HBA adapters for connecting internal and / or external disk baskets; hardware RAID controllers are not needed and are even contraindicated;
• one or more cache synchronization interfaces in a dual-controller configuration; There are several options: SAS, InfiniBand, Fiber Channel (FC), Ethernet; interface duplication is possible; in the single-controller version they are not needed;
• interfaces for connecting to SAN and / or NAS: Ethernet, InfiniBand, FC; possible direct connection to hosts (clients) via SAS;
• SAS / SATA disks (HDD) of large (3.5 ") or small (2.5") form factor; any suitable model from any manufacturer, without restrictions; SSD are also supported, but in the context of video surveillance they are not interesting;
• server platform suitable for installing the above equipment.

To connect a large number of disks it is supposed to use external disk shelves (baskets) connected via SAS. It is recommended to use internal and external disk baskets with support for hot-swappable disks.

Of course, there is a compatibility sheet, a list of recommended and tested equipment.

Deployment options

RAIDIX offers two deployment options: single- and dual-controller. In the first version, the RAIDIX software is installed on one physical server acting as a storage controller. The disks are combined into a fail-safe RAID, but the server itself and some of its components form single points of failure. This may be acceptable for non-critical tasks.

Dual-controller configuration involves installing RAIDIX software on two identical physical servers, each of which becomes a storage controller. These can be separate server platforms, or a single platform with two server nodes (nodes, blades). Both controllers are physically connected to a single disk pool hosted on internal and external disk baskets. RAIDIX integrates two servers into a fail-safe active-active cluster, the controller cache is synchronized across dedicated interfaces.

In normal mode, the load is evenly distributed to two controllers - half of the volumes created on the disk array are served by one controller, the other half - by the second controllers. If for some reason one of the controller nodes fails, the entire load automatically switches to the "surviving" controller without interruption and data loss. This solution eliminates the presence of single points of failure and is suitable for critical projects that are sensitive to downtime.

AIC HA401-LB2 solution can be cited as a good example of a server platform for dual-controller RAIDIX storage configuration. This is a 4U platform for high-availability storage servers (cluster-in-a-box) with two identical server nodes, duplicated power supplies and a 24 HDD 3.5 ”internal disk cage with hot-swappable storage. Each server node supports two Xeon processors, up to 2TB of RAM and up to 6 PCIe expansion slots. This is enough to deploy a very productive and capacious storage on several hundred disks. This platform can be called one of the recommended, it is successfully used in many projects based on RAIDIX.

Such an approach makes it possible to create the optimal storage for any project, with the necessary volumes and performance. Volumes can vary from 12 (possible and less) to hundreds of disks per storage system. In this case, the required performance is determined by the selected Intel Xeon processors, the amount of RAM, the bandwidth of interfaces and adapters - you can lay them with a margin to increase the disk capacity. We choose what is needed and nothing superfluous - maximum flexibility and no overpayments.

Key features

Today, RAIDIX is a universal solution, suitable for file storage, and for virtualization, and for other tasks, with all copes quite well. However, initially the product was developed for media storage - post-production video, in particular in the film industry.

For these guys, the deciding factors were:

• Fast sequential (streaming) operations for working with heavy uncompressed video. The streams are large, they need to be recorded without loss and read without brakes.
• Large usable capacity. Volumes are large, standard solutions for RAID-10 are too expensive, we need solutions with checksums a la RAID-5/6.
• No drawdown of performance in case of disk failure when the array is in a degraded state. Standard RAID arrays in their normal state, when all disks are alive, can produce the required performance, however, when degraded, it drops dramatically, which can lead to frame loss, creepy brakes and inability to work.
• High resiliency (availability). The absence of single points of failure, the guarantee of data integrity with the simultaneous failure of at least two disks in the array. The cost of data loss, which was very long and expensively removed and mounted, is too high, it is unacceptable to lose this data.

It was possible to satisfy these requirements only with the help of branded Hi-End storage systems that cost “blue” money. Therefore, we had to look for an alternative, and a product was developed, which later grew into an independent solution, became a RAIDIX, and now a lot of films are being mounted on studios around the world. Common storage tasks on RAIDIX are also successful. At the same time, its value is quite acceptable, one might say, folk.

RAIDIX for video surveillance

Main advantages

It is easy to draw parallels between media tasks and video surveillance and come to the conclusion that the requirements for storage infrastructure are the same: everything that is relevant for media storage is also important for video archives. Therefore, RAIDIX-based storage systems are optimal for video surveillance projects.

I / O processing to the disk subsystem and calculation of checksums for redundancy in RAIDIX is carried out by means of computing resources of Intel Xeon processors; SAS HBA adapters without RAID functions are used to connect the disks. The proprietary RAIDIX algorithms, using internal instructions from CPUs, perform these tasks much faster and more efficiently than hardware RAID controllers and ASIC devices. As a result, we get the capabilities necessary for a large infrastructure of video surveillance:

• Fast sequential (streaming) operations for processing data from a large number of IP cameras. For example, hundreds and thousands of cameras with FullHD resolution and above, with a frequency of 25 fps, medium or high complexity of the scene.
• Large usable capacity due to the ability to work effectively with large RAID groups on 12-24 disks, ensuring the processing of a simultaneous failure of up to two or more disks. At the same time large-capacity disks can be successfully used - 6-10TB.
• No drawdown of performance when taking out disks due to the power of Xeon processors and an effective approach to using their instructions. The data from the failed screws is very fast and imperceptible for end users of storage resources calculated from checksums.
• Support for dual-controller storage configuration, eliminating the presence of single points of failure.

Additional features

In addition, in the arsenal of RAIDIX there are the following unique and useful for the field of video surveillance "buns":

• In addition to standard RAID levels 10, 5, 6, RAID 7.3 and N + M are supported, which provide up to three failures (RAID-7.3) and more (RAID-N + M, N disks — usable capacity, M disks — can crash) at once.
• The ability to recover hidden data corruption on disks - protection against silent data corruption.
• Pre-emptive reconstruction. Storage monitors the performance of all disks. In case of malfunctions, individual disks can continue to work, but they can slow down so much that recovery of their data from checksums is faster than reading them. In this case, the record will remain unchanged, but instead of reading from the problem disks, the data will be forcibly calculated from the checksums of the array for optimal performance.
• Partial reconstruction. If the disk is temporarily disconnected after it is reconnected, only the missing part of the data will be restored. This can be useful when disconnecting disks by mistake, losing the connection of disks, moving disks between the shelves, disconnecting the entire shelves during maintenance (with a corresponding number of shelves and distributing RAID groups among them).

The truth of the possibilities stated above can be seen by deploying RAIDIX on a test bench. To do this, you need to request a demo version from us and conduct testing. Of course, this assumes that the hardware necessary for installing the RAIDIX software is available.

Active projects on RAIDIX

For more convincing, I propose to move from theory to practice. In addition to many installations in the media sector and solving general-purpose storage tasks, storage infrastructures have been successfully built based on RAIDIX in several large video surveillance projects.

At the moment, at the implementation stage, there is a large project in the territory of the Russian Federation for video surveillance throughout the city, in which 4 dual-controller RAIDIX storage systems were used for storage organization, with 200 HDDs of 6TB each. Another serious RAIDIX-based video surveillance project is being developed for a European casino: a dual-controller storage system for 300 HDDs of 10TB each. We will be able to talk about them in subsequent publications, after the introduction of systems into commercial operation.

Today we have three large successful video surveillance projects in South Korea. All three projects concern road safety and the recording of violations, implemented within the whole city. The largest of these is presented in the next section.

Project for 2000 IP cameras in Korea

Initial data

Yongin is one of the fastest growing megacities in South Korea, equal in area to Seoul, the city’s population is about a million people. To fix traffic violations and ensure security, the authorities installed 2,000 surveillance cameras in the city. Equipment for recording and storing video was located in 5 server racks and occupied the whole room. At the same time, the cameras shot low-resolution videos, which made the work of the law enforcement bodies much more difficult, since it was not possible to distinguish the car number or person in the image of such quality. Video recordings were kept for less than a month, and the police were limited in resources when investigating offenses - after this period the cases were closed, no fines were reported to the budget.



To solve this problem, the city administration has planned to record and store video in HD / FullHD resolution. Camera recording mode - 24/7 (around the clock) with a frequency of 16-25 frames per second. The archive retention period is at least one month. Such an approach significantly increases the video stream speed and data volume, therefore, requires a significant increase in performance (throughput) and storage capacity, which stores the video archive.

TK requirements for storage

Initially installed storage equipment with the task clearly could not cope, so there was a need to replace or upgrade. As a result, the following requirements were made for the new storage infrastructure:

• fast sequential recording and reading, elimination of delays and frame loss with simultaneous recording of 2000 HD / FullHD streams;
• no drawdown of the performance of the disk array in case of failure of the number of disks allowed by the configuration (array degradation)
• high reliability and fault tolerance, automatic failover of any element of the system;
• optimal price / quality ratio;
• simplicity and ease of implementation and maintenance;
• high-quality technical support;
• high density of equipment;
• high usable storage capacity sufficient to store the video archive from all cameras with a depth of at least one month;
• possibility of scaling.

Solution structure

As potential solutions, various options for brand and budget storage were considered. The first turned out to be too expensive, the second did not meet the stated requirements. The optimal solution among all the considered options that satisfy all the requirements of the project was RAIDIX-based storage systems. Below are the composition and structure of the implemented repository and the project as a whole.

In addition to the storage system, the IT infrastructure of the project included:

• 15 physical rack servers on the Intel platform running Windows Server for the deployment of VMS-servers (video surveillance software);
• 2 Mellanox InfiniBand (IB) switches with 40 Gbit / s port capacity for connecting VMS servers to the storage system.

To deploy the storage, a 4U rack-mount AIC HA401-CP2 server platform was used with two server nodes and an internal basket on a 24 ″ 3.5 ”disk — an analog of a newer platform AIC HA401-LB2. This platform is ideal for deploying a dual-controller RAIDIX fault-tolerant configuration, since each computing (server) node is a separate server, with its motherboard, processors, RAM, network interfaces and HBA adapters (for connecting disk baskets). Power supplies are duplicated and are common to all nodes. Thus, due to the duplication of all necessary components, single points of failure of the platform as a whole are eliminated.

Below is a block diagram of the solution.

Accessories

The hardware configuration of server storage controllers is identical, each controller includes:

• One Intel Xeon E5-2609 processor (4 cores at 2.4 GHz). Not the most recent and powerful "percent", but it was quite enough.
• 128GB of RAM - many disks and RAID groups require a large amount of RAM.
• 2 Mellanox ConnectX-3 2-port InfiniBand adapters, 56 Gb / s transfer rate per port. 2 ports for connecting to IB switches, 2 ports for synchronizing the controller cache - 2 crossover connections between server nodes.
• One Broadcom 9207-4i4e SAS HBA adapter for connecting to 1 external (60 disk JBOD) and 1 internal disk basket (24 disks). 4 external (4e) and 4 SAS internal lines at 6 Gb / s, 1 external Mini-SAS port.
• One Broadcom 9207-8e HBA adapter for connecting three external disk shelves (3 JBOD for 60 drives, 2 shelves in cascade). 8 external (8e) 6 Gbit / s SAS lines, 2 external Mini-SAS ports.

In the project, 208 SAS HDD 3.5 '' 4TB 7200r / min corporate-class hard drives were used to achieve the capacity required for storing the video archive. In addition to the internal disk basket of the storage platform (on 24 HDD), 4 external disk shelves (JBOD) AIC XJ3000-4603S were used to accommodate such a number of disks:

• 4U form factor;
• capacity - 60 HDD 3,5 "'with vertical installation and hot replacement;
• 2 6Gbit / s SAS-expander, 4 Mini-SAS ports per expander;
• power supply duplication.

All disk basket baskets (internal and external) supported hot-swappable disks. The inner and two outer baskets were packed with disks completely (24 + 2x60 = 144HDD), according to the structural diagram, each of them is independently connected to one HBA-adapter port on each controller. The remaining 64 disks were equally divided between the remaining two external shelves (32 disks per shelf), as shown in the diagram, the shelves are connected in cascade (sequentially) - the first shelf to the HBA adapters of the controllers, the second - two ways to the first.

Expandability

The solution supports further vertical scaling. Since the two disk baskets are not completely clogged, it is possible to install up to 56 additional disks in them. At the same time it is better to put on controllers for an additional HBA-adapter (Broadcom 9207-8e) and connect the cascaded disk shelf to them directly. On the remaining free ports of new adapters (as long as they took one of the two Mini-SAS ports), you can hang another shelf for 60 disks.

In case of a lack of computing power, you should add one processor per controller and increase the amount of RAM. Thus, it is possible to increase the storage capacity and performance by 1.5 times with minimal investment - the existing platform is expanding vertically without the need to purchase new servers.

Array configuration

When configuring a disk array, a simple and logical approach was chosen. Each of the 15 VMS servers was given one volume (LUN), each of which was located on a separate RAID group of 12-15 disks. To create each disk group, a RAID 7.3 level was chosen, ensuring data integrity with up to three any drives simultaneously failing.

As a result, the total useful storage capacity was about 600TB.

Placement density

All the equipment necessary for organizing the storage system was placed in one rack, as shown in the figure below.



It should be noted that while a significant part of the rack space remained empty, the usable space occupied by the iron storage system is 20U (5 nodes of 4U each), the density of placement is rather high.

Conclusion

This project was implemented in 2014, at that time the solution was quite modern and technological. The system has been successfully functioning for the fourth year already and is coping with the tasks set; no modernization or expansion has been carried out and is not yet required.

Today, the equipment used in this project is outdated: new generations and models of processors, HBA adapters are released, disk volumes have grown to 8-10TB. Therefore, the capacity of the project can be reached and exceeded on a new iron with a density of 8U instead of 20U.

Thus, on modern hardware and RAIDIX software storage systems, you can successfully build efficient data storage for large video surveillance systems that can provide high density and volumes up to several petabytes per storage system, while guaranteeing high performance and fault tolerance for reasonable money.