Intellectualization of dome dome camera: automatic patrol, target selection and tracking
Automation of the dome PTZ camera control system (PTZ cameras) is an interesting and relevant task. With the concentration of situational centers and the introduction of video analytics, there is a need for intelligent algorithms that allow not only to analyze video from stationary (fixed) cameras, but also to direct a robotic camera at a target without operator intervention. The delay introduced by the digital video encoding and decoding subsystem limits the ability to remotely track a target using a PTZ camera and heightens the need for local automation of tracking. Our post Habru contains an overview of the main tasks of intellectualizing PTZ cameras, approaches to solving them and offers on the market.
Consider the main tasks solved in the automation of the control system of a PTZ camera:
As part of the patrol function, the PTZ camera cyclically “bypasses” the observation set by the operator, stops at each position for a specified time and broadcasts the video with the selected magnification. This feature is standard and built into almost all models of dome PTZ cameras. The advantage of patrols on the prepositions is the ability to cover a large area and get images in each position with good detail. The disadvantages of the function are the presence of a blind zone in all positions except the current one and a constant change in the background of the scene, which complicates the analysis by video analytics and the operator. In patrol mode, it is difficult to recognize slow scene changes during a short interval of a camera in each position. If the operator directs the camera to a certain position, then events occurring in other positions are not recorded in the video archive.
These deficiencies can be eliminated by installing viewing fixed cameras that completely cover the protected area. Then the PTZ camera is used solely to obtain a detailed image of targets detected by viewing cameras. The life of the PTZ camera also increases due to the decrease in its mechanical load.
')
Signal sources for automatic target selection can be: a) a survey fixed camera used in parallel with the dome; b) dome camera in patrol mode; c) other sensors, for example, radio wave or vibration sensors of the perimeter system. The video signal from a television or thermal imaging camera is processed by video analytics, which detects targets and determines their location for pointing a PTZ camera without operator intervention. An example of the installation, the implemented data approach is presented in Fig. 1. If several survey cameras with overlapping coverage areas are used, then multi-channel (multi-camera) video analytics is desirable. Especially important is the multichannel video analytics with the frequent appearance of targets. Re-detection of the target by each camera will result in inefficient use of PTZ cameras and tracking disruptions, which will make it difficult to retrospectively analyze the archive.
In the case when several targets are in the field of view of the surveillance system, and the number of PTZ cameras is limited, it is required to distribute tasks between PTZ cameras in an optimal way in terms of their importance. The algorithm can calculate the priority of the target, taking into account several criteria, such as: a) the location of the target (proximity to the protected milestone or the most important object); b) the time of tracking the object (for example, each target must be accompanied by a PTZ camera for at least 10 seconds, after which it is possible to switch to another target); c) the classification of human behavior (for example, the behavior of “idleness in the zone” may have a higher priority than the “entrance to the zone”). All found goals are placed in a prioritized queue for subsequent processing by an intelligent video surveillance system.
The algorithm should pick up targets from the prioritized queue in order of importance and distribute the targets between the available PTZ cameras, taking into account the relative position of the targets and the available cameras. An operator can intervene in the operation of the algorithm, sending commands to a PTZ camera using a joystick or a software interface (Fig. 4). In this case, the algorithm should use other PTZ cameras to track targets that the operator has ignored. On complex objects, it is necessary to use three-dimensional models of the protected object and camera coverage areas .
In the simplest case, the targeting algorithm can be implemented with the help of a multi-zone motion detection camera of an overview camera: the frame is divided into a number of zones, each of which is associated with prepositions of a PTZ camera. When a motion detector is triggered in the zone (Fig. 2), the PTZ camera is transferred to the corresponding preposition (Figure 4). The more zones you set when tuning, the more magnification you can get on a PTZ camera. The disadvantage of this approach is unstable operation in the presence of several targets and limiting the accuracy of the guidance associated with the selected assumptions of the PTZ camera.
On an object with a large observation space and a large number of cameras, an analytical transformation of the coordinates of the viewing camera into the coordinate system of the rotating camera is recommended without dividing the frames into zones (Fig. 3.4).
Better guidance can be obtained using professional video analytics . The connection between a viewing and a controlled camera is established through the global coordinate system of the real world, to which all cameras are attached . The accuracy of the transformation from the two-dimensional frame coordinate system to the three-dimensional space of the real world limits the approximation of the PTZ camera, since in case of a conversion error, at a high magnification, the object may be out of sight. Therefore, special requirements are placed on the video analytics of a survey camera: high-quality localization (segmentation) of the target and high-quality calibration are necessary to connect its coordinates with a rotating camera.
After a PTZ camera is aimed at a target, it is advisable to use tracking algorithms to display and record a complete video fragment of the target, followed by a PTZ camera. In the process of setting up the tracking algorithm, you have to find a compromise between the degree of magnification (and, consequently, the granularity) of the target and the frequency of displacements of the PTZ camera. The stronger the increase, the more often you have to move the camera.
Common PTZ cameras do not allow the camera to rotate smoothly at variable speeds. With a step shift position of the PTZ camera, the image is “twitching” and blurred. Therefore, a good tracking algorithm should minimize the number of camera shifts for a given zoom. The tracking algorithm should work correctly in the case of a temporary overlap between the goals, for example, if people are going towards each other (see video demo and slides about the algorithm ).
PTZ tracking of a target can be done in three ways: a) using a PTZ camera (self-tracking); b) using an overview camera (external tracking) and c) in a hybrid way. Each of the methods has its advantages and disadvantages, which we compare in a separate publication. The self-tracking algorithm is convenient in the case when the operator sets the target manually, and the overview camera is missing or does not see the target. The external tracking algorithm is more stable when there are several targets. For objects of a single visible size, tracking algorithms on a moving camera work worse than on a fixed camera, since in the latter case, the algorithm can better adapt to the fixed background. In theory, a hybrid method should provide the most stable tracking in all situations, but in systems we know it has not yet been implemented.
Tracking a target with a PTZ drive is a real-time, time-sensitive task. If the total video delay in the IP network exceeds 500 ms (half a second), then neither the operator nor the server-based video analytics can effectively control the camera. As a rule, about 300 ms is entered by the transmitting device (camera or encoder) and about 100 ms is entered by the VMS system decoding the video.
High-quality tracking of the object can be implemented in the local video processing before compression. In this case, the coordinates of the target can be calculated according to the survey or PTZ camera for 20-40 ms. Such a system can accompany fast-moving targets, such as a running man and a vehicle, at a good magnification.
Starting from version 1.02, the international standard ONVIF allows building unified solutions for automatic and manual control of PTZ cameras. In particular, the standard describes the commands for controlling and reading the position of a PTZ camera, the coordinate system, and also the format for transferring metadata about moving objects from a viewing camera to a video management system (VMS) and / or other devices for controlling a PTZ camera.
The use of PTZ intellectual functions in public places is limited by the capabilities of video analytics tracking. Today, there are no video analytics on the market that can accompany a person in a crowd without using a face detector on a viewing camera. If the resolution and viewing angle of the viewing camera allows you to use a face detector, then you can automate the guidance of a PTZ camera to more accurately recognize faces and record a detailed image. At the same time, it is necessary to implement a tracking system according to the data of the face detector in order to optimize the performance of the PTZ camera for the desired scenario, for example, to track one person or to quickly scan all persons in sight.
Most PTZ cameras on the market with Pelco D interfaces (for RS422 / 485 serial interface) or ONVIF (for IP networks) do not have feedback from the control system, in particular, it is impossible to query the current camera position and set the camera in absolute coordinates . This restriction does not allow the use of a PTZ camera for tracking the coordinates of a viewing camera.
The DSSL Trassir ActiveDome module of the company has a PTZ tracking function with analytical coordinate conversion. In the frame of the overview camera, an area is defined which, by means of a calibration procedure, creates a connection between the coordinates and the PTZ camera. According to information from the developer, the number of surveillance cameras in the video surveillance system can be unlimited and is associated with the size of the controlled area. For example, to provide a 360 ° view, it is recommended to install 4 viewing and one PTZ camera.
The iTV Intellect product can implement PTZ tracking using multi-zone motion detection of a viewing camera without automating the calibration process. To do this, you must perform the steps: 1) split the frame of the overview camera into multiple areas of motion detection; 2) program the corresponding pre-positions on the PTZ camera; 3) write a script that will set the PTZ camera in the preposition corresponding to the movement zone. For PTZ tracking in the conditions of movement of two or more targets, it is necessary to implement more complex logic using a script or an ActiveX component.
Our company is working on the implementation of PTZ-tracking with multi-zone motion detection and analytical coordinate transformation in IP-video server MagicBox. In the current version of the firmware of the device, the transfer of metadata with the coordinates of the goals and the PTZ drive control is carried out within the international standard ONVIF, which allows the external control logic of the PTZ camera to be implemented. The ONVIF Device Manager application, with which Habr is already familiar , illustrates the interaction of the ONVIF client with a PTZ camera and a video analytic service (Fig. 4).
Fig. 3. Tracking the target with built-in video analytics. Transfer 2D and 3D target coordinates in ONVIF metadata for automatic pointing of a PTZ camera. The letter M means that the target is moving. The letter S means that the target has stopped. The background of the target is moving (the leaves of the trees are moving).
Fig. 4. Manual and automatic control of a PTZ camera using ONVIF protocol through the ONVIF Device Manager.
Automatic control technologies for a robotic PTZ camera based on video analytics and other sensors are at an early stage of their development. VSS-systems of DSSL, ITV companies, as well as the stand-alone MagicBox device of the Aggregator and Synesis companies, allowing to automate the operation of a PTZ camera, are on the Russian market. It should be noted that there are promising areas for improving these products: a) implementation of algorithms for working with several targets with the help of several PTZ cameras in a single space of survey cameras; b) elaboration of the semi-automatic mode, for example, when the operator begins to monitor one target, the system should use free PTZ cameras to track other targets; c) simplification of the initial setup process (calibration) of the system and optimization of the user interface for operation in automatic and semi-automatic PTZ tracking modes.
 |  |
| Fig. 1. Experimental setup for autonomous PTZ tracking: MagicBox video analytic device, Pelco PTZ camera and CNB surveillance camera. | Fig. 2. Prepositional PTZ cameras, controlled by a zone motion detector. |
Automation tasks
Consider the main tasks solved in the automation of the control system of a PTZ camera:
1. Auto Patrol
As part of the patrol function, the PTZ camera cyclically “bypasses” the observation set by the operator, stops at each position for a specified time and broadcasts the video with the selected magnification. This feature is standard and built into almost all models of dome PTZ cameras. The advantage of patrols on the prepositions is the ability to cover a large area and get images in each position with good detail. The disadvantages of the function are the presence of a blind zone in all positions except the current one and a constant change in the background of the scene, which complicates the analysis by video analytics and the operator. In patrol mode, it is difficult to recognize slow scene changes during a short interval of a camera in each position. If the operator directs the camera to a certain position, then events occurring in other positions are not recorded in the video archive.
These deficiencies can be eliminated by installing viewing fixed cameras that completely cover the protected area. Then the PTZ camera is used solely to obtain a detailed image of targets detected by viewing cameras. The life of the PTZ camera also increases due to the decrease in its mechanical load.
')
2. Automatic target selection for PTZ tracking
Signal sources for automatic target selection can be: a) a survey fixed camera used in parallel with the dome; b) dome camera in patrol mode; c) other sensors, for example, radio wave or vibration sensors of the perimeter system. The video signal from a television or thermal imaging camera is processed by video analytics, which detects targets and determines their location for pointing a PTZ camera without operator intervention. An example of the installation, the implemented data approach is presented in Fig. 1. If several survey cameras with overlapping coverage areas are used, then multi-channel (multi-camera) video analytics is desirable. Especially important is the multichannel video analytics with the frequent appearance of targets. Re-detection of the target by each camera will result in inefficient use of PTZ cameras and tracking disruptions, which will make it difficult to retrospectively analyze the archive.
3. Automatic prioritization for detailing and tracking
In the case when several targets are in the field of view of the surveillance system, and the number of PTZ cameras is limited, it is required to distribute tasks between PTZ cameras in an optimal way in terms of their importance. The algorithm can calculate the priority of the target, taking into account several criteria, such as: a) the location of the target (proximity to the protected milestone or the most important object); b) the time of tracking the object (for example, each target must be accompanied by a PTZ camera for at least 10 seconds, after which it is possible to switch to another target); c) the classification of human behavior (for example, the behavior of “idleness in the zone” may have a higher priority than the “entrance to the zone”). All found goals are placed in a prioritized queue for subsequent processing by an intelligent video surveillance system.
4. Automatic PTZ camera selection
The algorithm should pick up targets from the prioritized queue in order of importance and distribute the targets between the available PTZ cameras, taking into account the relative position of the targets and the available cameras. An operator can intervene in the operation of the algorithm, sending commands to a PTZ camera using a joystick or a software interface (Fig. 4). In this case, the algorithm should use other PTZ cameras to track targets that the operator has ignored. On complex objects, it is necessary to use three-dimensional models of the protected object and camera coverage areas .
5. Automatic PTZ camera guidance
In the simplest case, the targeting algorithm can be implemented with the help of a multi-zone motion detection camera of an overview camera: the frame is divided into a number of zones, each of which is associated with prepositions of a PTZ camera. When a motion detector is triggered in the zone (Fig. 2), the PTZ camera is transferred to the corresponding preposition (Figure 4). The more zones you set when tuning, the more magnification you can get on a PTZ camera. The disadvantage of this approach is unstable operation in the presence of several targets and limiting the accuracy of the guidance associated with the selected assumptions of the PTZ camera.
On an object with a large observation space and a large number of cameras, an analytical transformation of the coordinates of the viewing camera into the coordinate system of the rotating camera is recommended without dividing the frames into zones (Fig. 3.4).
Better guidance can be obtained using professional video analytics . The connection between a viewing and a controlled camera is established through the global coordinate system of the real world, to which all cameras are attached . The accuracy of the transformation from the two-dimensional frame coordinate system to the three-dimensional space of the real world limits the approximation of the PTZ camera, since in case of a conversion error, at a high magnification, the object may be out of sight. Therefore, special requirements are placed on the video analytics of a survey camera: high-quality localization (segmentation) of the target and high-quality calibration are necessary to connect its coordinates with a rotating camera.
6. Auto tracking
After a PTZ camera is aimed at a target, it is advisable to use tracking algorithms to display and record a complete video fragment of the target, followed by a PTZ camera. In the process of setting up the tracking algorithm, you have to find a compromise between the degree of magnification (and, consequently, the granularity) of the target and the frequency of displacements of the PTZ camera. The stronger the increase, the more often you have to move the camera.
Common PTZ cameras do not allow the camera to rotate smoothly at variable speeds. With a step shift position of the PTZ camera, the image is “twitching” and blurred. Therefore, a good tracking algorithm should minimize the number of camera shifts for a given zoom. The tracking algorithm should work correctly in the case of a temporary overlap between the goals, for example, if people are going towards each other (see video demo and slides about the algorithm ).
PTZ tracking of a target can be done in three ways: a) using a PTZ camera (self-tracking); b) using an overview camera (external tracking) and c) in a hybrid way. Each of the methods has its advantages and disadvantages, which we compare in a separate publication. The self-tracking algorithm is convenient in the case when the operator sets the target manually, and the overview camera is missing or does not see the target. The external tracking algorithm is more stable when there are several targets. For objects of a single visible size, tracking algorithms on a moving camera work worse than on a fixed camera, since in the latter case, the algorithm can better adapt to the fixed background. In theory, a hybrid method should provide the most stable tracking in all situations, but in systems we know it has not yet been implemented.
Delay effect
Tracking a target with a PTZ drive is a real-time, time-sensitive task. If the total video delay in the IP network exceeds 500 ms (half a second), then neither the operator nor the server-based video analytics can effectively control the camera. As a rule, about 300 ms is entered by the transmitting device (camera or encoder) and about 100 ms is entered by the VMS system decoding the video.
High-quality tracking of the object can be implemented in the local video processing before compression. In this case, the coordinates of the target can be calculated according to the survey or PTZ camera for 20-40 ms. Such a system can accompany fast-moving targets, such as a running man and a vehicle, at a good magnification.
Standards support
Starting from version 1.02, the international standard ONVIF allows building unified solutions for automatic and manual control of PTZ cameras. In particular, the standard describes the commands for controlling and reading the position of a PTZ camera, the coordinate system, and also the format for transferring metadata about moving objects from a viewing camera to a video management system (VMS) and / or other devices for controlling a PTZ camera.
Lively Scenes
The use of PTZ intellectual functions in public places is limited by the capabilities of video analytics tracking. Today, there are no video analytics on the market that can accompany a person in a crowd without using a face detector on a viewing camera. If the resolution and viewing angle of the viewing camera allows you to use a face detector, then you can automate the guidance of a PTZ camera to more accurately recognize faces and record a detailed image. At the same time, it is necessary to implement a tracking system according to the data of the face detector in order to optimize the performance of the PTZ camera for the desired scenario, for example, to track one person or to quickly scan all persons in sight.
Special requirements for PTZ camera
Most PTZ cameras on the market with Pelco D interfaces (for RS422 / 485 serial interface) or ONVIF (for IP networks) do not have feedback from the control system, in particular, it is impossible to query the current camera position and set the camera in absolute coordinates . This restriction does not allow the use of a PTZ camera for tracking the coordinates of a viewing camera.
Review of solutions on the market
The DSSL Trassir ActiveDome module of the company has a PTZ tracking function with analytical coordinate conversion. In the frame of the overview camera, an area is defined which, by means of a calibration procedure, creates a connection between the coordinates and the PTZ camera. According to information from the developer, the number of surveillance cameras in the video surveillance system can be unlimited and is associated with the size of the controlled area. For example, to provide a 360 ° view, it is recommended to install 4 viewing and one PTZ camera.
The iTV Intellect product can implement PTZ tracking using multi-zone motion detection of a viewing camera without automating the calibration process. To do this, you must perform the steps: 1) split the frame of the overview camera into multiple areas of motion detection; 2) program the corresponding pre-positions on the PTZ camera; 3) write a script that will set the PTZ camera in the preposition corresponding to the movement zone. For PTZ tracking in the conditions of movement of two or more targets, it is necessary to implement more complex logic using a script or an ActiveX component.
Our company is working on the implementation of PTZ-tracking with multi-zone motion detection and analytical coordinate transformation in IP-video server MagicBox. In the current version of the firmware of the device, the transfer of metadata with the coordinates of the goals and the PTZ drive control is carried out within the international standard ONVIF, which allows the external control logic of the PTZ camera to be implemented. The ONVIF Device Manager application, with which Habr is already familiar , illustrates the interaction of the ONVIF client with a PTZ camera and a video analytic service (Fig. 4).
Fig. 3. Tracking the target with built-in video analytics. Transfer 2D and 3D target coordinates in ONVIF metadata for automatic pointing of a PTZ camera. The letter M means that the target is moving. The letter S means that the target has stopped. The background of the target is moving (the leaves of the trees are moving).
Fig. 4. Manual and automatic control of a PTZ camera using ONVIF protocol through the ONVIF Device Manager.
Conclusion
Automatic control technologies for a robotic PTZ camera based on video analytics and other sensors are at an early stage of their development. VSS-systems of DSSL, ITV companies, as well as the stand-alone MagicBox device of the Aggregator and Synesis companies, allowing to automate the operation of a PTZ camera, are on the Russian market. It should be noted that there are promising areas for improving these products: a) implementation of algorithms for working with several targets with the help of several PTZ cameras in a single space of survey cameras; b) elaboration of the semi-automatic mode, for example, when the operator begins to monitor one target, the system should use free PTZ cameras to track other targets; c) simplification of the initial setup process (calibration) of the system and optimization of the user interface for operation in automatic and semi-automatic PTZ tracking modes.
Source: https://habr.com/ru/post/125496/
All Articles