Ivideon launched cloud video storage
Some artists cannot stand the sight of their creations when they are finished. And I am my biggest fan. OW Grant (Interstate 60)
The Ivideon team has been working on the remote storage service for archived video records in its own cloud for more than a year (I don’t like this word). Why so long, technical details and how to get free access to it for Habr's readers - you will learn under the cut.
')
There are several well-known platforms for creating cloud storage. For example, the same OpenStack, which, by the way, we also supported. Nevertheless, they have completely realized their decision for themselves. Why did we do that? To answer this question you need to understand the specifics of storing a remote archive of video surveillance. The fact is that almost all cloud platforms are designed primarily for the predominance of read operations on the record. This is partly why incoming traffic to the clouds is free, and outgoing costs money. For such a platform, in most cases it is not so important in what order the recorded files are stored on disks. Although work on the optimization is certainly underway.
The situation is different with video surveillance system storage. First of all, there is an obvious significant predominance of write operations on read operations. People view no more than 1 - 2% of all archived records. And sometimes even less.
Despite the fact that the recording is made by the motion detector, with a large number of clients the data flow is constantly. And everything would be fine if it were not for the limited amount of hard drives. At some point, the oldest user entries need to be deleted in order to make room for new ones. And here problems with ordinary clouds already begin.
Imagine the average user according to our statistics per day can generate about 1200 video recordings of observation of different duration from one camera. For the video to be available to the user not after the end of the entire recording, but, for example, in a minute, it is required to divide each recording into small portions, which should be stored in the cloud. That is, files may not be 1200, but significantly more. Suppose they are 2500.
Now imagine that on a single physical hard disk, there are files for a week from 200 users. And at some point, the procedure of archive rotation is launched - deleting the oldest files recorded in three days. It is necessary to delete about 1.5 million files at once! Certainly - the cloud will work. Slowly, but it will work. Only hard drives for a long time will not sustain such a load, and as you know, in cloud storage it is one of the most expensive elements. And we get that cloud storage will be a very expensive service for the average user. And we have always strived to give people not only the highest quality, but also affordable service from all that is on the market.
The Ivideon cloud has a hierarchical structure and is created as an independent element of the entire Ivideon platform. That is, it can be developed and tested separately from the entire service as a whole. We strive to provide this approach throughout our development so that relatively independent groups of developers can work on individual modules. As a result, we are able to increase the speed of developing elements of our platform, attracting new developers to the project.
As frontend we use a group of servers with a large amount of RAM. Currently, each of these servers contains about 96 GB of memory.
The task of this server is the accumulation of ready fragments from the cameras for several minutes. All these fragments are written exclusively in memory.
About 80 - 100 MB are allocated for each camera. The server receives many video frames from different cameras. As soon as the fragment finishes, it is immediately queued for sending to the backend storage. The memory is specially allocated more than the size of one fragment, so that at the time of sending the completed fragment there is a place to write a new one.
What are backends like? This is a group of servers with 36 disks each, which can be located in one data center or in several, depending on the required degree of reliability of storage. Let's call these servers nodes. Each fragment can be duplicated on a given number of nodes, depending on the setting of the user's tariff plan. And to distribute the load on the nodes are used sharding mechanisms. The maximum speed of a physical record in a node is calculated and controlled so that the amount of data transferred per unit of time does not exceed this parameter.
From a software point of view, we support the nodes OpenStack, web-dav, nfs, etc. But to store the archive in our data centers, we use our own software implementation of the node. In this case, the connection with it is carried out via the internal HTTP-base protocol. In fact, an almost instantaneous transfer of an already completed fragment and its writing to the node is performed. As well as entering the fragment data into the internal database. The base is mongoDB, installed on separate servers.
But that is not all. To solve the problem of deleting a large amount of old records and reducing disk fragmentation as a result of this deletion in our node implementation, we developed our own file system. Of course, it sounds quite loud, but there’s really nothing complicated about it. Each fragment is written to disk no longer as a separate file, but combined in series with the previous ones, forming one large file with records within an hour. In the following articles we will try to tell about it in more detail.
At the moment when it is necessary to rotate the archive and delete the data for several days, the node is temporarily marked as not ready for recording and new fragments are distributed among other nodes. This is done so that during deletion no entry is made to the node to prevent fragmentation.
Then the uninstall process itself starts. In which you need to delete not 7500 small files from each user, but only 72, since each file is a record in one hour.
Here I did not talk about many of the intricacies of the work of our platform, for example, associated with different tariff plans. Someone keeps a week archive, and someone month. Also, I did not focus on the specific details of working with well-known elements used in our platform. For example, MongoDB. And this is the topic for the next article.
I hope, thanks to this, the presentation turned out clear and at the same time useful.
I just want to add that, as always, we only use Linux in our data centers and optimize our server software for it.
And finally. We tried very hard to conduct a comprehensive and load testing of our remote storage platform. And nevertheless, we launch it in Beta mode and set a “restraining” tariff so that the growth in load is controlled from our side. But if someone from Habr's reader wants to help us, by taking part in testing, write in a personal note, specify the e-mail of your personal account and we will deposit the required amount of money into your account so that the use of Ivideon cloud storage is free for you.
The Ivideon team has been working on the remote storage service for archived video records in its own cloud for more than a year (I don’t like this word). Why so long, technical details and how to get free access to it for Habr's readers - you will learn under the cut.
')
Why not use existing cloud storage?
There are several well-known platforms for creating cloud storage. For example, the same OpenStack, which, by the way, we also supported. Nevertheless, they have completely realized their decision for themselves. Why did we do that? To answer this question you need to understand the specifics of storing a remote archive of video surveillance. The fact is that almost all cloud platforms are designed primarily for the predominance of read operations on the record. This is partly why incoming traffic to the clouds is free, and outgoing costs money. For such a platform, in most cases it is not so important in what order the recorded files are stored on disks. Although work on the optimization is certainly underway.
The situation is different with video surveillance system storage. First of all, there is an obvious significant predominance of write operations on read operations. People view no more than 1 - 2% of all archived records. And sometimes even less.
Despite the fact that the recording is made by the motion detector, with a large number of clients the data flow is constantly. And everything would be fine if it were not for the limited amount of hard drives. At some point, the oldest user entries need to be deleted in order to make room for new ones. And here problems with ordinary clouds already begin.
Imagine the average user according to our statistics per day can generate about 1200 video recordings of observation of different duration from one camera. For the video to be available to the user not after the end of the entire recording, but, for example, in a minute, it is required to divide each recording into small portions, which should be stored in the cloud. That is, files may not be 1200, but significantly more. Suppose they are 2500.
Now imagine that on a single physical hard disk, there are files for a week from 200 users. And at some point, the procedure of archive rotation is launched - deleting the oldest files recorded in three days. It is necessary to delete about 1.5 million files at once! Certainly - the cloud will work. Slowly, but it will work. Only hard drives for a long time will not sustain such a load, and as you know, in cloud storage it is one of the most expensive elements. And we get that cloud storage will be a very expensive service for the average user. And we have always strived to give people not only the highest quality, but also affordable service from all that is on the market.
How we implemented our storage
The Ivideon cloud has a hierarchical structure and is created as an independent element of the entire Ivideon platform. That is, it can be developed and tested separately from the entire service as a whole. We strive to provide this approach throughout our development so that relatively independent groups of developers can work on individual modules. As a result, we are able to increase the speed of developing elements of our platform, attracting new developers to the project.
As frontend we use a group of servers with a large amount of RAM. Currently, each of these servers contains about 96 GB of memory.
The task of this server is the accumulation of ready fragments from the cameras for several minutes. All these fragments are written exclusively in memory.
About 80 - 100 MB are allocated for each camera. The server receives many video frames from different cameras. As soon as the fragment finishes, it is immediately queued for sending to the backend storage. The memory is specially allocated more than the size of one fragment, so that at the time of sending the completed fragment there is a place to write a new one.
What are backends like? This is a group of servers with 36 disks each, which can be located in one data center or in several, depending on the required degree of reliability of storage. Let's call these servers nodes. Each fragment can be duplicated on a given number of nodes, depending on the setting of the user's tariff plan. And to distribute the load on the nodes are used sharding mechanisms. The maximum speed of a physical record in a node is calculated and controlled so that the amount of data transferred per unit of time does not exceed this parameter.
From a software point of view, we support the nodes OpenStack, web-dav, nfs, etc. But to store the archive in our data centers, we use our own software implementation of the node. In this case, the connection with it is carried out via the internal HTTP-base protocol. In fact, an almost instantaneous transfer of an already completed fragment and its writing to the node is performed. As well as entering the fragment data into the internal database. The base is mongoDB, installed on separate servers.
But that is not all. To solve the problem of deleting a large amount of old records and reducing disk fragmentation as a result of this deletion in our node implementation, we developed our own file system. Of course, it sounds quite loud, but there’s really nothing complicated about it. Each fragment is written to disk no longer as a separate file, but combined in series with the previous ones, forming one large file with records within an hour. In the following articles we will try to tell about it in more detail.
At the moment when it is necessary to rotate the archive and delete the data for several days, the node is temporarily marked as not ready for recording and new fragments are distributed among other nodes. This is done so that during deletion no entry is made to the node to prevent fragmentation.
Then the uninstall process itself starts. In which you need to delete not 7500 small files from each user, but only 72, since each file is a record in one hour.
Summary
Here I did not talk about many of the intricacies of the work of our platform, for example, associated with different tariff plans. Someone keeps a week archive, and someone month. Also, I did not focus on the specific details of working with well-known elements used in our platform. For example, MongoDB. And this is the topic for the next article.
I hope, thanks to this, the presentation turned out clear and at the same time useful.
I just want to add that, as always, we only use Linux in our data centers and optimize our server software for it.
And finally. We tried very hard to conduct a comprehensive and load testing of our remote storage platform. And nevertheless, we launch it in Beta mode and set a “restraining” tariff so that the growth in load is controlled from our side. But if someone from Habr's reader wants to help us, by taking part in testing, write in a personal note, specify the e-mail of your personal account and we will deposit the required amount of money into your account so that the use of Ivideon cloud storage is free for you.
Source: https://habr.com/ru/post/147894/
All Articles