Automatic downloading of the list of prohibited sites from the Roskomnadzor service
Platform: Windows XP / Vista / 7
Language: Perl
The service created by Roskomnadzor bears a very long, difficult to understand name: “The Unified Registry of Domain Names, web site indexes and network addresses that allow identifying Internet sites containing information whose distribution is prohibited in the Russian Federation” therefore, we will further call it for simplicity the Roskomnadzor service.
Task
To organize automatic downloading of the lists of prohibited sites from the Roskomnadzor resource. The system must be implemented within a single host, the downloaded files must be rotated with a given depth.
')
Lyrics
Regarding the list of prohibited sites, it has already been said a lot and there is no point in discussing this topic again.
However, the daily downloading of the list of prohibited sites, despite the obvious meaninglessness of this lesson, has become a necessity for many Internet providers. This is done mainly in order to avoid claims from Roskomnadzor. Naturally, the first thing that comes to mind is to automate the download process. Unfortunately, the automation method on the Roskomnadzor website is described rather vaguely, and technical support is practically absent. In addition, in the case of process automation, working with the service is based on the SOAP protocol, which also does not add joy.
On this occasion, I would like to present one of the possible ways to organize automatic downloading of the list of prohibited sites. It is possible that someone also had such a problem, and I hope my article will help those people who are currently looking for its solution.
I must say that a number of articles have already been written on this topic, including on Habré, the methods presented in these articles, for the most part, did not allow achieving full automation. Somewhere it was necessary to manually form a request for a list, somewhere to run scripts, etc. In my case, the system works fully automatically, I only sometimes glance if everything is normal. Frankly, in order to raise the system, I used the article I found on Toster. It introduced Perl scripts, which I modified a little and added something.
Implementation of course turned out pretty rough, but quite working. All managed to place on the same host. This is an old computer from Windows XP accounting, in which Rutoken was plugged in, plugged into the network and locked in a rack. Accordingly, the directory into which the archives are downloaded from the Roskomnadzor service is shared to the local area network.
Now consider the implementation.
As I already said, Windows XP was used as the OS (which version does not matter), and Windows Vista and Windows 7 are also suitable for this purpose.
Initially, we believe that the only purpose of the host is to work with the registry of prohibited sites.
The acquisition of electronic signatures is a separate topic for conversation. You can purchase it from various companies that have the appropriate license, how to do it — usually it’s worth finding out in the company itself. In our case, the electronic signature was acquired from the company “CRYPTO-PRO”, therefore, I will describe this particular case.
The purchased package included: a signature carrier (Rutoken) with a cryptographic key, an ES certificate, a certificate for technical support, and a software CD.
When the OS is installed and the host is ready for work, the first thing to do is to install the necessary software for working with the ES.
First, install the drivers for Rutoken (those that were on the software CD did not fit). You can find them on the manufacturer's website (section Rutoken for Windows) .
Download drivers and install. If everything is done correctly, after Rutoken is inserted into the USB port, the LED will light on it. (The program is accordingly called “Rutoken Control Panel”).
Next, install the program Crypto-PRO. This should be done with the connected Rutoken.
The certificate is valid for one year, then it must be renewed. In this case, you must remove the old certificate and connect a new one. Certificate Information
can be found in “Start-> Crypto-Pro-> Certificates”
If there are difficulties in this place, then it is best to call technical support, there they will explain everything quite correctly.
In my case, I renewed an outdated certificate. To do this, I went to the "Certificates" section, then
“Certificates-current user-> Personal-> Registry-> Certificates” , and deleted the old certificate manually, inserted Rutoken and reloaded the machine. The current certificate was picked up automatically.
To verify that the certificate is installed correctly, you can manually sign the file using the Crypto-ARM program, which is included in the package.
To do this, install the program and create an arbitrary, preferably non-empty file.
Turn on the Crypto-ARM, then “Signature-> Sign”, select the file and go through the dialogue, at the end choose the certificate and click OK. If done correctly. Then in the directory specified in the dialog, the file filename.sig will appear. This file is a detached electronic signature.
The Crypto-AWP program does not participate in the system of automatically uploading lists of prohibited sites.
The most important thing is that we need the csptest.exe utility, (an ES check can be done with it), which is part of Crypto-PRO.
This is a console utility for creating ES. Works only under Windows.
According to the developers, this utility is needed exclusively for tests and cannot be used in a combat system, since she has incomplete capabilities. For normal operation, another utility is used that can work on a Linux server, but you have to pay for it separately.
Practice has shown that csptest.exe is quite suitable for solving the problem posed.
Work with the service "Roskomnadzor" occurs under the SOAP protocol.
Algorithm next
Once every 5 minutes we check if the download was performed for the current day (does the archive exist with the correct name)
If yes, do nothing.
If not, start downloading.
Create a current request to upload a list of prohibited sites
Create a signature file
We send a request to the server and get the request code (or an error message if the request is correct)
If the request is correct, wait 5 minutes (the request is processed 1-2 minutes) and send the request code to the server
We accept the archive with the list of files in binary mode
If the request is incorrect, we get the error code and do nothing.
The unloading system consists of the following:
4 files, of which 3 perl scripts and one .bat file
Files: make_request.pl , get_register.pl , rotation.pl , roscomnadzor.bat
The appointment of scripts.
make_request.pl - creates a request to the Roskomnadzor service
get_register.pl - sends requests to the Roskomnadzor service and processes the results
rotation.pl - rotates downloaded archives with a given depth
roscomnadzor.bat - synchronizes the work of scripts and utilities
Create a working directory in the root of the C: // drive (or sometimes a drive). I have it called roscomnadzor.
Inside it we create 2 subdirectories: source and result.
Scripts should be placed in the source directory, the result directory will be located in the result directory, i.e. Archives with lists of prohibited sites.
After placing the scripts, you must register the paths to the files from the root directory (or put in the config, if you want to make everything beautiful).
To perform the procedure, you must run the file roscomnadzor.bat
Listing roscomnadzor.bat
The make_request.pl script is responsible for creating the request, the request file is created in the source directory
The download request represents an xml file of the following form:
Listing make_request.pl
To create a signature file, we use the very same utility csptest.exe. It is called from a bat file.
Where:
-my - Specify the owner of the key;
-in - Specifies which file to sign. If the file is not in the folder with csptest, then you need to specify the full path .;
-out - Specifies the name of the signature file;
The get_register.pl script is responsible for sending requests and processing the results .
Listing get_register.pl
The result of the script execution (provided the server is available) is an archive with the following name: reestr_yyyy_mm_dd.zip
Even with a low download frequency, sooner or later all the space on the hard disk of the machine will be entertaining, which can lead to the most disastrous consequences. In order to keep the system working for a long time, it is necessary to organize rotation, i.e. as new archives are downloaded, old ones will be gradually deleted.
For rotation, the rotation.pl script
The rotation depth determines how many archives will be in the result directory at the same time (for a given logic, for what time interval in the past relative to the current day)
Listing rotation.pl
To run on a schedule using the standard Windows Task Scheduler
schtasks ( http://www.windowsfaq.ru/content/view/83/57 )
The time interval through which roscomnadzor.bat will be launched must be longer than the waiting time for a request from the server
I started the system in mid-March, while everything works without fail.
Thanks for attention. I hope my article was useful to you.
Language: Perl
The service created by Roskomnadzor bears a very long, difficult to understand name: “The Unified Registry of Domain Names, web site indexes and network addresses that allow identifying Internet sites containing information whose distribution is prohibited in the Russian Federation” therefore, we will further call it for simplicity the Roskomnadzor service.
Task
To organize automatic downloading of the lists of prohibited sites from the Roskomnadzor resource. The system must be implemented within a single host, the downloaded files must be rotated with a given depth.
')
Lyrics
Regarding the list of prohibited sites, it has already been said a lot and there is no point in discussing this topic again.
However, the daily downloading of the list of prohibited sites, despite the obvious meaninglessness of this lesson, has become a necessity for many Internet providers. This is done mainly in order to avoid claims from Roskomnadzor. Naturally, the first thing that comes to mind is to automate the download process. Unfortunately, the automation method on the Roskomnadzor website is described rather vaguely, and technical support is practically absent. In addition, in the case of process automation, working with the service is based on the SOAP protocol, which also does not add joy.
On this occasion, I would like to present one of the possible ways to organize automatic downloading of the list of prohibited sites. It is possible that someone also had such a problem, and I hope my article will help those people who are currently looking for its solution.
I must say that a number of articles have already been written on this topic, including on Habré, the methods presented in these articles, for the most part, did not allow achieving full automation. Somewhere it was necessary to manually form a request for a list, somewhere to run scripts, etc. In my case, the system works fully automatically, I only sometimes glance if everything is normal. Frankly, in order to raise the system, I used the article I found on Toster. It introduced Perl scripts, which I modified a little and added something.
Implementation of course turned out pretty rough, but quite working. All managed to place on the same host. This is an old computer from Windows XP accounting, in which Rutoken was plugged in, plugged into the network and locked in a rack. Accordingly, the directory into which the archives are downloaded from the Roskomnadzor service is shared to the local area network.
Now consider the implementation.
As I already said, Windows XP was used as the OS (which version does not matter), and Windows Vista and Windows 7 are also suitable for this purpose.
Initially, we believe that the only purpose of the host is to work with the registry of prohibited sites.
1. Qualified detached electronic signature (ES) and software (software) for working with it
The acquisition of electronic signatures is a separate topic for conversation. You can purchase it from various companies that have the appropriate license, how to do it — usually it’s worth finding out in the company itself. In our case, the electronic signature was acquired from the company “CRYPTO-PRO”, therefore, I will describe this particular case.
The purchased package included: a signature carrier (Rutoken) with a cryptographic key, an ES certificate, a certificate for technical support, and a software CD.
When the OS is installed and the host is ready for work, the first thing to do is to install the necessary software for working with the ES.
First, install the drivers for Rutoken (those that were on the software CD did not fit). You can find them on the manufacturer's website (section Rutoken for Windows) .
Download drivers and install. If everything is done correctly, after Rutoken is inserted into the USB port, the LED will light on it. (The program is accordingly called “Rutoken Control Panel”).
Next, install the program Crypto-PRO. This should be done with the connected Rutoken.
The certificate is valid for one year, then it must be renewed. In this case, you must remove the old certificate and connect a new one. Certificate Information
can be found in “Start-> Crypto-Pro-> Certificates”
If there are difficulties in this place, then it is best to call technical support, there they will explain everything quite correctly.
In my case, I renewed an outdated certificate. To do this, I went to the "Certificates" section, then
“Certificates-current user-> Personal-> Registry-> Certificates” , and deleted the old certificate manually, inserted Rutoken and reloaded the machine. The current certificate was picked up automatically.
To verify that the certificate is installed correctly, you can manually sign the file using the Crypto-ARM program, which is included in the package.
To do this, install the program and create an arbitrary, preferably non-empty file.
Turn on the Crypto-ARM, then “Signature-> Sign”, select the file and go through the dialogue, at the end choose the certificate and click OK. If done correctly. Then in the directory specified in the dialog, the file filename.sig will appear. This file is a detached electronic signature.
The Crypto-AWP program does not participate in the system of automatically uploading lists of prohibited sites.
The most important thing is that we need the csptest.exe utility, (an ES check can be done with it), which is part of Crypto-PRO.
This is a console utility for creating ES. Works only under Windows.
According to the developers, this utility is needed exclusively for tests and cannot be used in a combat system, since she has incomplete capabilities. For normal operation, another utility is used that can work on a Linux server, but you have to pay for it separately.
Practice has shown that csptest.exe is quite suitable for solving the problem posed.
2. Organization of automatic download
Work with the service "Roskomnadzor" occurs under the SOAP protocol.
Algorithm next
Once every 5 minutes we check if the download was performed for the current day (does the archive exist with the correct name)
If yes, do nothing.
If not, start downloading.
Create a current request to upload a list of prohibited sites
Create a signature file
We send a request to the server and get the request code (or an error message if the request is correct)
If the request is correct, wait 5 minutes (the request is processed 1-2 minutes) and send the request code to the server
We accept the archive with the list of files in binary mode
If the request is incorrect, we get the error code and do nothing.
The unloading system consists of the following:
4 files, of which 3 perl scripts and one .bat file
Files: make_request.pl , get_register.pl , rotation.pl , roscomnadzor.bat
The appointment of scripts.
make_request.pl - creates a request to the Roskomnadzor service
get_register.pl - sends requests to the Roskomnadzor service and processes the results
rotation.pl - rotates downloaded archives with a given depth
roscomnadzor.bat - synchronizes the work of scripts and utilities
System deployment
Create a working directory in the root of the C: // drive (or sometimes a drive). I have it called roscomnadzor.
Inside it we create 2 subdirectories: source and result.
Scripts should be placed in the source directory, the result directory will be located in the result directory, i.e. Archives with lists of prohibited sites.
After placing the scripts, you must register the paths to the files from the root directory (or put in the config, if you want to make everything beautiful).
Principle of operation
To perform the procedure, you must run the file roscomnadzor.bat
Listing roscomnadzor.bat
if not exist "D:\roscomnadzor\result\reestr_%Date:~6,4%_%Date:~3,2%_%Date:~0,2%.zip" ( rem perl D:\roscomnadzor\source\make_request.pl rem < csptest>csptest.exe -sfsign -sign -detached -add -in < >request.txt -out < >request.txt.sig -my < >@example.ru rem perl D:\roscomnadzor\source\get_register.pl ren D:\roscomnadzor\result\reestr.zip "reestr_%Date:~6,4%_%Date:~3,2%_%Date:~0,2%.zip" perl D:\roscomnadzor\source\rotation.pl ) 1. Creating a request file
The make_request.pl script is responsible for creating the request, the request file is created in the source directory
The download request represents an xml file of the following form:
<?xml version="1.0" encoding="windows-1251"?> <request> <requestTime>yyyy-mm-ddT115:00:00.000+04:00</requestTime> <operatorName> </operatorName> <inn> </inn> <ogrn> </ogrn> <email>company@cmail.ru</email> </request> Listing make_request.pl
use POSIX qw(strftime); my $date = strftime "%Y-%m-%d", localtime; # unlink('< >request.txt'); unlink('< >request.txt.sig'); # xml $request='<?xml version="1.0" encoding="windows-1251"?><request><requestTime>'.$date.'T115:00:00.000+04:00</requestTime><operatorName>COMPANY NAME</operatorName><inn>1111111111</inn><ogrn>1111111111111</ogrn><email>company@example.ru</email></request>'; # my $filename = '< >request.txt'; open(my $fh, '>', $filename) or die " '$filename' $!"; print $fh $request; close $fh; 2. Creating a signature file
To create a signature file, we use the very same utility csptest.exe. It is called from a bat file.
D:\"Program files"\"Crypto Pro"\CSP\csptest.exe -sfsign -sign -detached -add -in < >request.txt -out < >request.txt.sig -my < >@example.ru Where:
-my - Specify the owner of the key;
-in - Specifies which file to sign. If the file is not in the folder with csptest, then you need to specify the full path .;
-out - Specifies the name of the signature file;
3. Dialogue with the Roskomnadzor service
The get_register.pl script is responsible for sending requests and processing the results .
Listing get_register.pl
use MIME::Base64; use SOAP::Lite; use POSIX qw(strftime); #----------------------------------------------------------------------- # , $key1=0; $key2=0; if (open(FP, '< >request.txt')) {$key1=1;} if (open(FP, '< >request.txt.sig')) {$key2=1;} # - if( ($key1==1)&($key2==1)) { #---------------------------- # open(FIL,"< >request.txt"); while ($line = <FIL>) { $ln.=$line; } close (FIL); $req=$ln; #---------------------------- # open(FIL1,"< >request.txt.sig"); while ($line1 = <FIL1>) { $ln1.=$line1; } close (FIL1); $sig=$ln1; #---------------------------- # wsdl- $soap = SOAP::Lite->service('http://vigruzki.rkn.gov.ru/services/OperatorRequest/?wsdl'); $r = $soap->getLastDumpDate(); # @r = $soap->sendRequest($req, $sig); $code = $r[2]; # 5 sleep 1, print "$_\n" for 1..300; #300 @r = $soap->getResult($code); #------------------------- # ( ) #my $date = strftime "%Y-%m-%d_at_%H-%M-%S", localtime; #my $date = strftime "%Y-%m-%d", localtime; my $filename = '< >reestr.zip'; open(my $fh, '>', $filename) or die " '$filename' $!"; binmode $fh; print $fh decode_base64($r[1]); close $fh; } The result of the script execution (provided the server is available) is an archive with the following name: reestr_yyyy_mm_dd.zip
4. Rotation of results
Even with a low download frequency, sooner or later all the space on the hard disk of the machine will be entertaining, which can lead to the most disastrous consequences. In order to keep the system working for a long time, it is necessary to organize rotation, i.e. as new archives are downloaded, old ones will be gradually deleted.
For rotation, the rotation.pl script
The rotation depth determines how many archives will be in the result directory at the same time (for a given logic, for what time interval in the past relative to the current day)
Listing rotation.pl
use POSIX qw(strftime); use Time::Local; # $birthtime =timelocal(localtime()); $interval = 0 + # 0 0 * 60 + # 0 0 * 60 * 60 + # 0 50 * 60 * 60 * 24; # 20 $then = $birthtime - $interval; my $date = strftime "%Y_%m_%d", localtime($then); # unlink('< >reestr_'.$date.'.zip'); 5. Scheduled Run
To run on a schedule using the standard Windows Task Scheduler
schtasks ( http://www.windowsfaq.ru/content/view/83/57 )
The time interval through which roscomnadzor.bat will be launched must be longer than the waiting time for a request from the server
I started the system in mid-March, while everything works without fail.
Thanks for attention. I hope my article was useful to you.
Source: https://habr.com/ru/post/224901/
All Articles