130 thousand surveillance cameras - how to make them work?
Hi, Habr! We would like to thank you again for the excellent feedback; we will give detailed answers to a number of your questions, including a detailed description of the infrastructure of our system. We ourselves thought soon to make such a post, but since you, too, expressed interest in this subject, we have a little forced the process.
Under the cut - immediately to the point.
The system architecture is built on a modular principle , which is the standard for systems of this scale. But there is one significant difference: these modules in the system are not just a lot, but a lot. Each module performs separate highly specialized tasks related to receiving video images and information from city and departmental information systems (IS), controlling access to the processed data, providing photo and video images to ECDD users and various external consumers, including residents of the city.
The modules closely interact with each other on the basis of various technologies (mainly JSON, REST API and SOAP). The modules themselves are implemented in various languages ​​(Java, C #, Java Script and others), using frameworks (ASP.NET Web API, WCF, NLog, Entity Framework, MySQL ADO.NET managed drivers, Unity 3, EPPlus, Json.NET, EmitMapper, DotNetZip, jQuery, knockoutjs, Moment.js, underscore.js and so on).
')
All this software diversity operates under various operating systems (Windows Server, SUSE Linux Enterprise Server, CentOS and others) in the VMware vSphere 5 virtualization environment.
The system architecture is shown in the diagram:
All modules have their own mechanisms for fault tolerance and balancing. Cluster technologies, various options for balancing HTTP requests based on nginx are used for this, as well as queue and priority management at the application level. In addition, all modules are combined according to the principle of weak connectivity, which significantly increases the possibilities for development and modification of the system as a whole.
This approach allows us to make the system not only scalable, but also very flexible in terms of implementing new technological processes or making changes to existing ones, since each module can develop independently and support for backward compatibility of the module is imperative.
The video core, which provides for receiving and processing video streams , consists of several hundreds of video servers running on dedicated virtual machine resources running Linux, ensuring reception of video traffic from more than 145 thousand cameras, coders and other video streaming systems. The traffic is received via the RTSP protocol (video in H.264 format), the traffic receiving pattern can be used differently - on an ongoing basis with a customized recording depth (retention period), or on request if you simply need to watch the video. Such a flexible approach allows saving both server resources and reducing the workload of communication channels.
TCP is mainly used as a transport layer protocol, but in some cases UDP can also be used. Video servers perform several tasks:
- Primary: receiving a video stream, recording it, and then issuing streaming or archived video to retransmission servers or long-term storage of a portion of the archived video on a separate dedicated storage;
- A number of additional: monitoring the availability of a video source, monitoring the receipt and compliance of a video stream with specified parameters (fps, bitrate, packet loss), as well as through specialized data communication buses, information from video servers goes to the service delivery monitoring system for subsequent accounting and transmission to operational services.
Management of video servers is carried out through a specialized HTTP API that allows you to fully automate the work through control systems. The operation of video servers is controlled through Zabbix, using additional internal mechanisms of video servers.
Restrimming - a retransmission server is a link between the video core (performing the primary video reception and recording) and users (essentially the main consumers of video content). Built-in mechanisms for the reencapsulation and distribution of streaming video allow for the retransmission of live and archived video content to PCs and portable devices (tablets, phones). Restrimming can also provide streaming video content over RTSP for additional systems, for example, video analytics systems or transcoding management systems - after all, users sometimes need a “compressed” video stream if they suddenly “dipped” the channel, and watch the video very much :) Also users can use the special mode of viewing as a slideshow. Relay servers operate in a cluster and support dynamic backup mode, as well as isolating the video core from possible surges in user-generated traffic.
The user interface - a web interface (front-end) provides authorized access for several tens of thousands of registered users to the video surveillance system and provides a wide range of different functionalities available from anywhere in the intracity network, and in some cases through closed dedicated Internet resources.
Work is provided in the environment of various operating systems (Windows, MacOS, Linux) on all major browsers, which greatly simplifies the organization of workplaces. It also supports work with a number of mobile devices, for example, based on iOS.
The functionality is quite diverse - from the usual viewing of streaming and archival video (via flash player on PC, and native iOS tools) and controlling PTZ camera functions, search engines with links to different layers of cartography and integration with urban systems data, and to use a flexible role model, the formation of a personal representation of the user interface and built-in mechanisms for user training.
Front-end uses technologies such as RequireJS, knockout, lodash, leaflet and others. The back-end is built according to a modular principle, which allows for ensuring the necessary level of redundancy and scaling of components, using a configuration management system. Software and technologies: Java / Tomcat, MariaDB, RabbitMQ and several others.
Integration gateway is a set of modules of subsystems that provide isolation of external consumers of information from the system core. Issues various information to external systems after passing authorization and taking into account specified powers (i.e. available data volume and functionality). There are three main logical components:
The use of these components allows external systems not only to access data and form their own information visualization interface, but also to implement already prepared components as simply as possible.
To provide users with access to cameras of the video surveillance system, there is a whole range of interrelated components that provide user authentication and authorization, prioritization of access to PTZ control functions, logging of user actions and interaction of individual modules.
The system supports two types of authentication: using the user accounts of the ECDC and using a single city control system of access to resources (in fact, this is an SSO option for citywide IP). At the same time, for a single physical user, it is possible to share different types of authentication. The key authentication information of the ECDD users is stored in Active Directory, and all additional information is stored in the Microsoft SQL database.
All passwords, of course, are transmitted between the modules in encrypted form. Or they are not transmitted at all, as we also can :)
The granting of access rights and priorities is implemented on the basis of a role model for all modules, which allows providing access not only to video surveillance cameras, but also to individual functions of the system. Currently, there are more than 100 different authorities in the system: access to live broadcast, the ability to view and export a photo and video archive, PTZ control, make changes to individual system registers and directories, create a schedule for taking pictures, create patrol routes , access from mobile devices and much more. The system is constantly evolving, new user groups with their tasks are connected, and the addition of new powers to the system occurs, in fact, on the fly.
In addition, there are several levels of priorities for different user groups and system services, which allows avoiding conflicts between different user groups and transparently intercepting PTZ camera control, providing control in patrol modes, or moving the field of view on a schedule.
In order to quickly and simply provide permissions for tens of thousands of users and manage them, the system uses various mechanisms: assignment of permissions to groups of users, use of templates with predefined typical permissions, smart groups (when based on specified rules new users are automatically distributed into groups ). Similar mechanisms are used to manage camera registries, but the distribution into groups is mainly implemented according to geographical principle, type of service, or belonging to a departmental video surveillance system.
Access control is carried out almost in real time, and any change in the composition of powers immediately affects the user. In addition, token-based validation session mechanisms are used to control the lifetime of links to live video streams. By the way, it was the absence of this mechanism in the test service for the residents of the city that made it possible to get access to the “expired” links to the video streams.
The system will record all user actions and interaction of modules at all levels, which will allow an analysis of any situation. The logging system records the date and time of the event, what actions the user performed (up to pressing the interface buttons), what tasks the scheduler performed in the system, what ended the information exchange and much more. To date, the system has recorded more than 10 billion records of the implementation of "technological" processes, each of which consists of several separate actions being recorded. Every day, DIT employees receive statistics that allow users to determine user activity in various sections: which portals they used, the number of live viewings, which camera groups had the largest number of requests, etc.
As a “cherry on the cake,” video surveillance system in numbers:
Number of cameras: more than 145 thousand;
Number of users: more than 10 thousand;
Network video traffic volume: about 120 Gbit / s;
Storage capacity : 20 PB.
The official page of the system here . You can see the camera specifications, learn how to act in the event of an accident that could get into the lens, and view several cameras in public places using the Window to City service .
Read also in our blog on Habré:
» Habraeffect for 130,000 Moscow cameras
» Information technology feeds more than 750 thousand people in Moscow
» Blog of the Department of Information Technology of the city of Moscow on" Habrahabr "
Thanks for attention!
Under the cut - immediately to the point.
The system architecture is built on a modular principle , which is the standard for systems of this scale. But there is one significant difference: these modules in the system are not just a lot, but a lot. Each module performs separate highly specialized tasks related to receiving video images and information from city and departmental information systems (IS), controlling access to the processed data, providing photo and video images to ECDD users and various external consumers, including residents of the city.
The modules closely interact with each other on the basis of various technologies (mainly JSON, REST API and SOAP). The modules themselves are implemented in various languages ​​(Java, C #, Java Script and others), using frameworks (ASP.NET Web API, WCF, NLog, Entity Framework, MySQL ADO.NET managed drivers, Unity 3, EPPlus, Json.NET, EmitMapper, DotNetZip, jQuery, knockoutjs, Moment.js, underscore.js and so on).
')
All this software diversity operates under various operating systems (Windows Server, SUSE Linux Enterprise Server, CentOS and others) in the VMware vSphere 5 virtualization environment.
The system architecture is shown in the diagram:
All modules have their own mechanisms for fault tolerance and balancing. Cluster technologies, various options for balancing HTTP requests based on nginx are used for this, as well as queue and priority management at the application level. In addition, all modules are combined according to the principle of weak connectivity, which significantly increases the possibilities for development and modification of the system as a whole.
This approach allows us to make the system not only scalable, but also very flexible in terms of implementing new technological processes or making changes to existing ones, since each module can develop independently and support for backward compatibility of the module is imperative.
Description of the main components of the system
The video core, which provides for receiving and processing video streams , consists of several hundreds of video servers running on dedicated virtual machine resources running Linux, ensuring reception of video traffic from more than 145 thousand cameras, coders and other video streaming systems. The traffic is received via the RTSP protocol (video in H.264 format), the traffic receiving pattern can be used differently - on an ongoing basis with a customized recording depth (retention period), or on request if you simply need to watch the video. Such a flexible approach allows saving both server resources and reducing the workload of communication channels.
TCP is mainly used as a transport layer protocol, but in some cases UDP can also be used. Video servers perform several tasks:
- Primary: receiving a video stream, recording it, and then issuing streaming or archived video to retransmission servers or long-term storage of a portion of the archived video on a separate dedicated storage;
- A number of additional: monitoring the availability of a video source, monitoring the receipt and compliance of a video stream with specified parameters (fps, bitrate, packet loss), as well as through specialized data communication buses, information from video servers goes to the service delivery monitoring system for subsequent accounting and transmission to operational services.
Management of video servers is carried out through a specialized HTTP API that allows you to fully automate the work through control systems. The operation of video servers is controlled through Zabbix, using additional internal mechanisms of video servers.
Restrimming - a retransmission server is a link between the video core (performing the primary video reception and recording) and users (essentially the main consumers of video content). Built-in mechanisms for the reencapsulation and distribution of streaming video allow for the retransmission of live and archived video content to PCs and portable devices (tablets, phones). Restrimming can also provide streaming video content over RTSP for additional systems, for example, video analytics systems or transcoding management systems - after all, users sometimes need a “compressed” video stream if they suddenly “dipped” the channel, and watch the video very much :) Also users can use the special mode of viewing as a slideshow. Relay servers operate in a cluster and support dynamic backup mode, as well as isolating the video core from possible surges in user-generated traffic.
The user interface - a web interface (front-end) provides authorized access for several tens of thousands of registered users to the video surveillance system and provides a wide range of different functionalities available from anywhere in the intracity network, and in some cases through closed dedicated Internet resources.
Work is provided in the environment of various operating systems (Windows, MacOS, Linux) on all major browsers, which greatly simplifies the organization of workplaces. It also supports work with a number of mobile devices, for example, based on iOS.
The functionality is quite diverse - from the usual viewing of streaming and archival video (via flash player on PC, and native iOS tools) and controlling PTZ camera functions, search engines with links to different layers of cartography and integration with urban systems data, and to use a flexible role model, the formation of a personal representation of the user interface and built-in mechanisms for user training.
Front-end uses technologies such as RequireJS, knockout, lodash, leaflet and others. The back-end is built according to a modular principle, which allows for ensuring the necessary level of redundancy and scaling of components, using a configuration management system. Software and technologies: Java / Tomcat, MariaDB, RabbitMQ and several others.
Integration gateway is a set of modules of subsystems that provide isolation of external consumers of information from the system core. Issues various information to external systems after passing authorization and taking into account specified powers (i.e. available data volume and functionality). There are three main logical components:
- The service interaction component is the HTTP API, through which the output of a given set of information (camera names, addresses and location coordinates, screenshots, and other accompanying information) is provided.
- Relay component - designed to minimize load on the video core and provide video streaming to external systems (broadcasting on flash is supported, as well as HLS for iOS devices). The video core and the restriction server have the functionality of a CDN , and this component works as an additional CDN link for the distribution of video content.
- The interface provision component is designed to be embedded into external systems as a ready-to-use interface for playing streaming and archival video information, for example, by embedding via iframe and simple customization (visual presentation settings — colors, necessary function buttons, etc.) through simple parameterized calls.
The use of these components allows external systems not only to access data and form their own information visualization interface, but also to implement already prepared components as simply as possible.
To provide users with access to cameras of the video surveillance system, there is a whole range of interrelated components that provide user authentication and authorization, prioritization of access to PTZ control functions, logging of user actions and interaction of individual modules.
The system supports two types of authentication: using the user accounts of the ECDC and using a single city control system of access to resources (in fact, this is an SSO option for citywide IP). At the same time, for a single physical user, it is possible to share different types of authentication. The key authentication information of the ECDD users is stored in Active Directory, and all additional information is stored in the Microsoft SQL database.
All passwords, of course, are transmitted between the modules in encrypted form. Or they are not transmitted at all, as we also can :)
The granting of access rights and priorities is implemented on the basis of a role model for all modules, which allows providing access not only to video surveillance cameras, but also to individual functions of the system. Currently, there are more than 100 different authorities in the system: access to live broadcast, the ability to view and export a photo and video archive, PTZ control, make changes to individual system registers and directories, create a schedule for taking pictures, create patrol routes , access from mobile devices and much more. The system is constantly evolving, new user groups with their tasks are connected, and the addition of new powers to the system occurs, in fact, on the fly.
In addition, there are several levels of priorities for different user groups and system services, which allows avoiding conflicts between different user groups and transparently intercepting PTZ camera control, providing control in patrol modes, or moving the field of view on a schedule.
In order to quickly and simply provide permissions for tens of thousands of users and manage them, the system uses various mechanisms: assignment of permissions to groups of users, use of templates with predefined typical permissions, smart groups (when based on specified rules new users are automatically distributed into groups ). Similar mechanisms are used to manage camera registries, but the distribution into groups is mainly implemented according to geographical principle, type of service, or belonging to a departmental video surveillance system.
Access control is carried out almost in real time, and any change in the composition of powers immediately affects the user. In addition, token-based validation session mechanisms are used to control the lifetime of links to live video streams. By the way, it was the absence of this mechanism in the test service for the residents of the city that made it possible to get access to the “expired” links to the video streams.
The system will record all user actions and interaction of modules at all levels, which will allow an analysis of any situation. The logging system records the date and time of the event, what actions the user performed (up to pressing the interface buttons), what tasks the scheduler performed in the system, what ended the information exchange and much more. To date, the system has recorded more than 10 billion records of the implementation of "technological" processes, each of which consists of several separate actions being recorded. Every day, DIT employees receive statistics that allow users to determine user activity in various sections: which portals they used, the number of live viewings, which camera groups had the largest number of requests, etc.
As a “cherry on the cake,” video surveillance system in numbers:
Number of cameras: more than 145 thousand;
Number of users: more than 10 thousand;
Network video traffic volume: about 120 Gbit / s;
Storage capacity : 20 PB.
The official page of the system here . You can see the camera specifications, learn how to act in the event of an accident that could get into the lens, and view several cameras in public places using the Window to City service .
Read also in our blog on Habré:
» Habraeffect for 130,000 Moscow cameras
» Information technology feeds more than 750 thousand people in Moscow
» Blog of the Department of Information Technology of the city of Moscow on" Habrahabr "
Thanks for attention!
Source: https://habr.com/ru/post/252639/
All Articles