Automation of processing video files from webcams using shell
It was necessary for the administration to organize in time video surveillance of some things on its own and meet the minimum funding. The task to automate this fell on the shoulders of the system administrator, that is - me.
Given: N - D-Link 2102 video cameras, a physical two-unit server for a video surveillance server and remote file storage.
The result should be the ability to let some users on the video surveillance server online and organize an archive of video recordings.
Under the cut there are several scripts that helped me a lot to understand how to write code better, why many things are needed and how they are solved, as well as to put things in order in my head and I really hope that they will help someone else.
The project was written quite difficult - so much I did not study the shell (bash) scripts - there was no need before.
But when the task is set and there is a solution algorithm in my head - all the scripts were reworked so that reading them in half a year I and my successor had no questions and no desire to rewrite everything from scratch.
UPD: post periodically updated.
')
The distribution is selected Ubuntu-Server 10.04. The server part for online viewing has become ZoneMinder. As I have been writing a lot of documentation for setting up Ubuntu and ZoneMinder, I would like to tell you about the automation of archive maintenance.
It was decided that as the main receiver of recordings from cameras (minute videos, it is logically justified from the point of view of minimizing video loss), the server on which ZoneMinder is spinning will be. The Samba-server is deployed on the server as the best and easiest way to solve the reception of video from cameras and the link between the cameras and the file storage.
All files related to the video archive are in / home / ipcamera / camname.
/ home / ipcamera / scripts - scripts that run on schedule in / etc / crontab once an hour.
/ home / ipcamera / out - mounted from the ball file storage where the processed video is added.
In dependencies, we have samba tools for mounting the partition where the archive is stored by samba, and bc, as a console calculator, plus codecs and avimerge.
Tasks:
1. Get video from cameras (one minute video)
2. Drain 60 minute clips into one.
3. Upload the received watch video to the file storage.
4. Remove from the video surveillance server all video that is older than 3 days.
5. Periodically delete from the archive video that is older than 3 months.
6. Give access to the archive to those who need and do not give to those who do not need.
At the network level, cameras are available for a video surveillance server, can be configured via a web interface, and file storage is also available.
All scripts are commented by max.
In the settings of the cameras, we set up the ntp server configured in the organization and the time of video recording, for example, from 9 to 18 in the settings of the cameras themselves.
Yes, I am aware of the quality of the recording and this issue is not considered in this article, as well as the legality of such self-assembled observation systems.
Configuring Samba and Zoneminder are not considered here either, although you cannot do without a piece of the Samba config, and it will.
The first config is a piece of the configuration scheduler.
It seems that indentation in formatting has disappeared in some places, please forgive me
Let's go in order:
The configuration file for all this disgrace.
Function description file to unload scripts and unify variables
The file that performs the merging of minute videos from cameras (this parameter in the cameras, unfortunately, cannot be changed, but it is advisable from the point of view of minimizing losses during recording and breaking the link between the camera and the receiving server)
Script that unloads all processed video to file storage.
Now - cleaning the video surveillance server from files older than 3 days
And for a snack
The fact is that sometimes the date flies on the cameras and they start writing in 2010, but by the time the files were created you can understand when they were actually created and rename them properly, it’s not enough for this procedure yet.
about resource intensity:
#uptime
17:49:27 up 2:21, 1 user, load average: 4.27, 4.25, 4.24
more CPU loaded than memory. This is ZoneMinder trying.
By logs:
On the order of 10 cameras, the execution time is now (HP Proliant DL560 G1, 1Gb of memory, Xeon 2x2188.804 MHz):
Creating file balls for each camera
A piece of samba config:
The cameras are now configured with a server by IP address and the name camname.
## TODO list.
1. Check whether the necessary packages are installed, it is possible to build in deb all this disgrace.
2. Monitor camera availability and alert those in charge.
3. Localization of scripts.
4. deb package and normal man / texinfo.
5. Probably it would be worthwhile to remove the output of many messages in &> / dev / null, but I haven’t figured it out yet.
6. Add code responsible for checking dependencies and requesting installation of them (connection with clause 4 is possible)
7. Since samba is integrated with the domain, it is possible to organize access by employee logins, but since cameras with domain logins do not work very well - does it make sense again.
8. Monitoring of liveliness of the link to them (ping) has not yet been implemented.
9. The monitoring of the fact that the camera was reset or simply did not receive the time via ntp and began to write video in the 2010th year is not implemented yet.
The last two stages - localization of variables to different languages, normal documentation and a package didn’t suffice me either, and does it make sense if the scripts are actually executed automatically and I go to the server well if once a month - when the whole algorithm works in my head , and, accordingly, checking for errors, the operator’s intervention became almost unnecessary, today the server had uptime of several months, and there was a first reboot for updating the kernel, samba and Kerberos, which are responsible for linking the server to Active Di rectory.
The only thing that confuses me is the overflow of the mail section, because for good all output should be suppressed, except for messages in the log file.
Thanks to that, I hope I helped you.
I am pleased to accept recipes for optimizing scripts and developing ideas further.
Sources are available here: on GitHub
Given: N - D-Link 2102 video cameras, a physical two-unit server for a video surveillance server and remote file storage.
The result should be the ability to let some users on the video surveillance server online and organize an archive of video recordings.
Under the cut there are several scripts that helped me a lot to understand how to write code better, why many things are needed and how they are solved, as well as to put things in order in my head and I really hope that they will help someone else.
The project was written quite difficult - so much I did not study the shell (bash) scripts - there was no need before.
But when the task is set and there is a solution algorithm in my head - all the scripts were reworked so that reading them in half a year I and my successor had no questions and no desire to rewrite everything from scratch.
UPD: post periodically updated.
')
The distribution is selected Ubuntu-Server 10.04. The server part for online viewing has become ZoneMinder. As I have been writing a lot of documentation for setting up Ubuntu and ZoneMinder, I would like to tell you about the automation of archive maintenance.
It was decided that as the main receiver of recordings from cameras (minute videos, it is logically justified from the point of view of minimizing video loss), the server on which ZoneMinder is spinning will be. The Samba-server is deployed on the server as the best and easiest way to solve the reception of video from cameras and the link between the cameras and the file storage.
All files related to the video archive are in / home / ipcamera / camname.
/ home / ipcamera / scripts - scripts that run on schedule in / etc / crontab once an hour.
/ home / ipcamera / out - mounted from the ball file storage where the processed video is added.
In dependencies, we have samba tools for mounting the partition where the archive is stored by samba, and bc, as a console calculator, plus codecs and avimerge.
Tasks:
1. Get video from cameras (one minute video)
2. Drain 60 minute clips into one.
3. Upload the received watch video to the file storage.
4. Remove from the video surveillance server all video that is older than 3 days.
5. Periodically delete from the archive video that is older than 3 months.
6. Give access to the archive to those who need and do not give to those who do not need.
At the network level, cameras are available for a video surveillance server, can be configured via a web interface, and file storage is also available.
All scripts are commented by max.
In the settings of the cameras, we set up the ntp server configured in the organization and the time of video recording, for example, from 9 to 18 in the settings of the cameras themselves.
Yes, I am aware of the quality of the recording and this issue is not considered in this article, as well as the legality of such self-assembled observation systems.
Configuring Samba and Zoneminder are not considered here either, although you cannot do without a piece of the Samba config, and it will.
The first config is a piece of the configuration scheduler.
It seems that indentation in formatting has disappeared in some places, please forgive me
#cat /etc/crontab 5 9-20 * * 1-6 root /home/ITC/ipcamera/scripts/split # 10 9-20 * * 1-6 root /home/ITC/ipcamera/scripts/upload # - 30 20 * * 1-6 root /home/ITC/ipcamera/scripts/clear # 0 21 * * 6 root /home/ITC/ipcamera/scripts/archieve # . 0 23 * * * root rm /var/spool/nullmailer/queue/* # – mail spooler . Let's go in order:
The configuration file for all this disgrace.
#cat scripts.conf # . camshare="/home/ipcamera/" # filename="*.avi" # currentdate=`date +%Y%m%d"/"%H` # , , smb, - . outfiles="out/" # Lock-, . lockfile=`basename $0`".lock" # . logfile="logfile" # tmpfile=`basename $0`".tmp" # upload_share_name="//filearchieve/web_camera_video" # upload_share_path="/home/ipcamera/out" # upload_share_user="ipcamerauser" # upload_share_passwd="ipcamerapassword" # share_mount_command="mount -t cifs" # , . Age=3 Function description file to unload scripts and unify variables
#cat functions.sh # . # ver 1.12. ### . . /home/ipcamera/scripts/scripts.conf # log_msg() { echo `date +%Y"/"%m"/"%d" "%T`" "`basename $0`": $1." >> $camshare$logfile } # - . lock_on() { # , , - . if [ -f $camshare$lockfile ]; then echo `date +%d"/"%m"/"%Y" "%T`" , $0 ? ." log_msg " , " exit else # - . touch $camshare$lockfile echo `date +%d"/"%m"/"%Y" "%T`" " fi } # . lock_off() { if [ -f $camshare$lockfile ]; then # . rm $camshare$lockfile echo `date +%d"/"%m"/"%Y" "%T`" ." else # , . echo `date +%d"/"%m"/"%Y" "%T`" ." fi } # . clear_tmp() { if [ -f $camshare$tmpfile ]; then # . rm $camshare$tmpfile echo `date +%d"/"%m"/"%Y" "%T`" $tmpfile ." else # . echo `date +%d"/"%m"/"%Y" "%T`" $tmpfile ." fi } # . upload_share_mount() { # , . test=`mount | grep "$upload_share_path"` if [ "$?" -eq "0" ] ; then # - . echo " $upload_share_path ." else # . `$share_mount_command $upload_share_name $upload_share_path -o user=$upload_share_user"%"$upload_share_passwd` # . if [ "$?" -eq "0" ] ; then # . echo " $upload_share_path ." else # . echo ": $upload_share_path . ." log_msg ": $upload_share_path . ." exit fi fi } # . upload_share_umount() { # , . test=`mount | grep "$upload_share_path"` if [ "$?" -eq "0" ] ; then # , . `umount "$upload_share_path"` echo " $upload_share_path ." else # , . echo " $upload_share_path ." fi } The file that performs the merging of minute videos from cameras (this parameter in the cameras, unfortunately, cannot be changed, but it is advisable from the point of view of minimizing losses during recording and breaking the link between the camera and the receiving server)
#cat split #!/bin/sh ## . #ver 1.0. # . . /home/ipcamera/scripts/functions.sh lock_on log_msg "" # , . find $camshare -type d | awk {'FS="/"} {print"/"$2"/"$3"/"$4"/"$5"/"$6"/"$7"/"$8}' | grep -v '/$' | grep -v '$outfiles' | grep -v $currentdate > $camshare$tmpfile # , , . for i in `cat $camshare$tmpfile` ; do cd $i ; [ -f $i.avi ] || avimerge -i `ls | sort` -o $i.avi ; rm -fr $i ; done clear_tmp log_msg "" lock_off Script that unloads all processed video to file storage.
#cat upload #!/bin/sh . /home/ipcamera/scripts/functions.sh ## , , . #ver 2.2. lock_on log_msg "" upload_share_mount # AA.avi, , . find "$camshare" -type f -name '[0-9][0-9].avi' | awk '{FS="/"} {print"/"$2"/"$3"/"$4"/"$5"/"$6"/"$7"/"$8}' | grep -v "/$" | grep -v "$outfiles" | grep -v "$currentdate" > "$camshare$tmpfile" # / - . for source in `cat $camshare$tmpfile` ; do city=`echo "$source" | awk -F / '{ print $5 }'` date=`echo "$source" | awk -F / '{ print $7 }'` fname=`echo "$source" | awk -F / '{ print $8 }'` [ -d "$camshare$outfiles$city" ] || mkdir "$camshare$outfiles$city" | echo " $city ." [ -d "$camshare$outfiles$city/$date" ] || mkdir "$camshare$outfiles$city/$date" | echo " $camshare$outfiles$city/$date ." [ -f "$camshare$outfiles$city/$date/$fname" ] || cp "$source" "$camshare$outfiles$city/$date/$fname" done clear_tmp upload_share_umount log_msg "" lock_off Now - cleaning the video surveillance server from files older than 3 days
#cat clear #!/bin/sh ## , 3 . #ver 1.0 # . . /home/ipcamera/scripts/functions.sh lock_on log_msg "" upload_share_mount # offset="172800" # 3 .# TODO unixdate=`date +%s` # . timediff=`echo "$unixdate"-"$offset" | bc` # offset'. # AA.avi, , . find "$camshare" -name '[0-9][0-9].avi' | awk '{FS="/"} {print"/"$2"/"$3"/"$4"/"$5"/"$6"/"$7"/"$8}' | grep -v '/$' | grep -v "$outfiles" | grep -v `date +%Y%m%d` > $camshare$tmpfile # . for source in `cat "$camshare$tmpfile"` ; do city=`echo "$source" | awk -F / '{ print $5 }'` # "". date=`echo "$source" | awk -F / '{ print $7 }'` # -"- "". fname=`echo "$source" | awk -F / '{ print $8 }'` # -"- " ". filedate=`ls -l --time-style=long-iso "$source" | awk '{print $6}'` # . unixfiledate=`date +%s -d"$filedate"` # . outfilename="$camshare$outfiles$city/$date/$fname" # . # , , timediff', - . if [ -f "$outfilename" ] ; then echo "$outfilename ." if [ "$unixfiledate" -lt "$timediff" ] ; then echo "$source ." rm "$source" else echo "$source" "$unixfiledate ." fi else echo " $outfilename ." fi done # . find "$camshare" -type d -empty | grep 'video' | xargs rm -fr {} upload_share_umount clear_tmp log_msg "" lock_off And for a snack
#cat archieve #!/bin/sh ## , . #ver 1.0. # . . /home/ipcamera/scripts/functions.sh lock_on log_msg "" upload_share_mount # . Year=`date +%Y` Month=`date +%m` Old=`echo "$Month"-"$Age"|bc` find "$camshare" -type d -name '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'| grep "$outfiles" > $camshare$tmpfile # . for source in `cat "$camshare$tmpfile"` ; do ddate=`echo "$source" | awk -F / '{ print $7 }'` Y=`echo $ddate | head -c4` M=`echo $ddate | head -c6 | tail -c2` D=`echo -n $ddate | tail -c2` if [ $Y -eq `date +%Y` ] ; then # echo "$source ." if [ "$M" -le "$Old" ] ; then # echo " $source , ." rm -rf $source else echo " $source ($Age ), ." fi else # echo " $source ." # TODO : mtime , 2010 . echo $source >> $camshare/2010.log fi done #TODO . upload_share_umount clear_tmp log_msg "" lock_off The fact is that sometimes the date flies on the cameras and they start writing in 2010, but by the time the files were created you can understand when they were actually created and rename them properly, it’s not enough for this procedure yet.
about resource intensity:
#uptime
17:49:27 up 2:21, 1 user, load average: 4.27, 4.25, 4.24
more CPU loaded than memory. This is ZoneMinder trying.
By logs:
On the order of 10 cameras, the execution time is now (HP Proliant DL560 G1, 1Gb of memory, Xeon 2x2188.804 MHz):
2011/10/29 20:30:01 clear: . 2011/10/29 20:30:11 clear: . 2011/10/29 21:00:01 archieve: . 2011/10/29 21:00:12 archieve: . 2011/10/31 09:05:01 split: . 2011/10/31 09:05:30 split: . 2011/10/31 09:10:02 upload: . 2011/10/31 09:10:16 upload: . Creating file balls for each camera
#cat mkshare #!/bin/bash ## . # Version name: 2.0. # functions.sh. # . . /home/ipcamera/scripts/functions.sh lock_on log_msg "" # - . if [ "$1" != "" ]; then mkdir "$camshare$1" # chmod 777 "$camshare$1" # # - samba . echo -e "\n[$1]\n comment = $1\n browseable = yes\n path = $camshare$1\n printable = no\n guest ok = yes\n read only =no\n create mask = 0700" >> /etc/samba/smb.conf # samba. service smbd restart # . echo "!!! IP Samba ACL, , samba : sudo service smbd restart" log_msg " $1 ." else # . echo " $0." fi A piece of samba config:
allow hosts = 10.0.0.1, 10.0.0.2 # , . [camname1] comment = camname1 browseable = yes path = /home/ipcamera/camname1 printable = no guest ok = yes read only = no The cameras are now configured with a server by IP address and the name camname.
## TODO list.
1. Check whether the necessary packages are installed, it is possible to build in deb all this disgrace.
2. Monitor camera availability and alert those in charge.
3. Localization of scripts.
4. deb package and normal man / texinfo.
5. Probably it would be worthwhile to remove the output of many messages in &> / dev / null, but I haven’t figured it out yet.
6. Add code responsible for checking dependencies and requesting installation of them (connection with clause 4 is possible)
7. Since samba is integrated with the domain, it is possible to organize access by employee logins, but since cameras with domain logins do not work very well - does it make sense again.
8. Monitoring of liveliness of the link to them (ping) has not yet been implemented.
9. The monitoring of the fact that the camera was reset or simply did not receive the time via ntp and began to write video in the 2010th year is not implemented yet.
The last two stages - localization of variables to different languages, normal documentation and a package didn’t suffice me either, and does it make sense if the scripts are actually executed automatically and I go to the server well if once a month - when the whole algorithm works in my head , and, accordingly, checking for errors, the operator’s intervention became almost unnecessary, today the server had uptime of several months, and there was a first reboot for updating the kernel, samba and Kerberos, which are responsible for linking the server to Active Di rectory.
The only thing that confuses me is the overflow of the mail section, because for good all output should be suppressed, except for messages in the log file.
Thanks to that, I hope I helped you.
I am pleased to accept recipes for optimizing scripts and developing ideas further.
Sources are available here: on GitHub
Source: https://habr.com/ru/post/134440/
All Articles