We work with the registry of prohibited resources

Automation of the registry of prohibited resources using C # , OpenSSL and filtering using RouterOS based on MikroTik equipment



Attention! Article amended according to recent changes on 19/03/2014

')

Introductory

In our work we will be guided by the operator's memo
The work consists of the following items:

1. Create request
2. Installing and configuring OpenSSL
3. Request Signature
4. Submitting a request and getting the result of processing the request
5. Result processing
6. Adding to the filter on MikroTikʻe

Create request

To submit a request for receiving an unloading from the registry, you must attach the request file in XML format . The file has the following form:

<?xml version="1.0" encoding="windows-1251"?> <request> <requestTime>2012-01-01T01:01:01.000+04:00</requestTime> <operatorName><![CDATA[ ]]></operatorName> <inn>1234567890</inn> <ogrn>1234567890123</ogrn> <email>email@email.ru</email> </request> 

• requestTime - the date and time of the request with the time zone;
• operatorName - full name of the carrier;
• inn - operator's TIN (10 digits for legal entities, 12 digits for individual entrepreneurs);
• ogrn - operator's OGRN (13 digits for legal entities, 15 digits for individual entrepreneurs);
• email - the email address of the technician responsible for using the unloading receipt mechanism; can
  be used for prompt feedback in case of technical issues or problems. Optional parameter.

The main thing is to get time in the format 2012-01-01T01: 01: 01.000 + 04: 00 and save the file with windows-1251 encoding.

C # request generation function code:

 public static String GeneratingRequest(String operatorName, String inn, String ogrn, String email) { String result = "<?xml version=\"1.0\" encoding=\"windows-1251\"?>"; result += "<request><requestTime>"; result += DateTime.Now.ToString("yyyy-MM-ddTHH:mm:ss.fffzzz"); result += "</requestTime><operatorName>"; result += "<![CDATA[" + operatorName + "]]>"; result += "</operatorName><inn>"; result += inn; result += "</inn><ogrn>"; result += ogrn; result += "</ogrn><email>"; result += email; result += "</email></request>"; return result; } 

Now save the file in windows-1251 encoding:

 String Request = GeneratingRequest(" ", "1234567890", "1234567890123", "email@email.ru") StreamWriter swRequest = new StreamWriter(@"C:\request.xml", false, Encoding.GetEncoding("Windows-1251")); swRequest.Write(Request); swRequest.Close(); 

After creating the request, we need to sign it. To do this, we use the open source cryptographic package for working with SSL / TLS - OpenSSL . Install and configure.

Openssl

Download the package from this site - slproweb.com . In my case, it was Win64 OpenSSL v1.0.1g .

Yes, this assembly requires the installed Visual C ++ 2008 Redistributables to work , which can be downloaded there.
Install. When installing, select the “The OpenSSL” binaries (/ bin) directory ” in the “ Select Additional Tasks ” dialog and there are no more tricks.
Next, go to the folder where you installed: C: / OpenSSL / bin and edit the file openssl.cfg . To the top of the file, add:

openssl_conf = openssl_def

In the end:
[openssl_def]
engines=engine_section

[engine_section]
gost=gost_section

[gost_section]
engine_id=gost
dynamic_path = C:/OpenSSL/bin/gost.dll
default_algorithms=ALL

Everything is almost working now. It remains to set up the environment variables:
OPENSSL_CONF = C: /OpenSSL/bin/openssl.cfg - full path to openssl.cfg
well, in PATH + = C: / OpenSSL / bin;

Now we need an EDS. You can purchase it in a trusted certification center . The key must be exported in PKCS # 12 format from a cryptocontainer to Windows using the utility P12FromGostCSP



The PKCS # 12 file must contain a certificate and a private key! You can check this with the command:

 openssl.exe pkcs12 -in C:/key.pfx -nodes 

Next, convert it to PEM . In OpenSSL, this is done like this (via the command line — it can request a password that protects the PKCS # 12 Key):

 openssl.exe pkcs12 -in C:/key.pfx -out C:/key.pem -nodes -clcerts 

All now we are able to sign our request.

Request Signature

You can sign the request via OpenSSL with the following command:

 openssl.exe smime -sign -in C:/request.xml -out C:/request.xml.sign -signer C:/key.pem -outform DER 

Implementing the C # file signature feature:

 public static Boolean SignRequest() { Boolean ret = true; String OpenSSLPath = @"C:\OpenSSL\bin"; String RequestPath = @"C:\request.xml"; String SignRequestPath = @"C:\request.xml.sign"; String KeyPEMPath = @"C:\key.pem"; try { Process cmdProcess = new Process(); /* *     *    PATH *    OpenSSL */ cmdProcess.StartInfo.WorkingDirectory = OpenSSLPath; cmdProcess.StartInfo.FileName = "openssl.exe"; cmdProcess.StartInfo.Arguments = String.Format("smime -sign -in {0} -out {1} -signer {2} -outform DER", RequestPath, SignRequestPath, KeyPEMPath); cmdProcess.StartInfo.CreateNoWindow = true; cmdProcess.StartInfo.WindowStyle = ProcessWindowStyle.Hidden; cmdProcess.Start(); // dimzon541 if (!cmdProcess.WaitForExit(5000)) { cmdProcess.Kill(); ret = false; } } catch (Exception) { ret = false; } return ret; } 

Now check out our work. Confirmation of authenticity of electronic signature : gosuslugi.ru , choose - Confirmation of authenticity of an electronic document. EA - disconnected, in PKCS # 7 format

You can check it using OpenSSL tools, but for this you need to extract the certificate of the publisher, which provided the EDS and convert it to the PEM format (you can also take the certificate here ):

 openssl x509 -inform der -in C:/most.cer -out C:/most.pem 

And check our signature:

 openssl.exe smime -verify -in C:/request.xml.sign -content C:/request.xml -CAfile C:/most.pem -inform DER -out C:/validrequest.xml 

As a result, we will see the line:

 Verification successful 

and in the validrequest.xml file will be the contents of request.xml .

We proceed directly to the request to dump the registry of prohibited resources.

Filing and unloading the registry

Everything is simple in manual: Request submission form , we send the request file and its signature ( C: /request.xml and C: /request.xml.sign )



If everything is normal, the result will be - the identifier of the request, with the help of it you can check the result of processing the request. As a rule, if everything is normal, then the registry will be unloaded in about 5 minutes in a ZIP format in the archive there will be two dump.xml files - a registry dump and its digital signature.

You can check the file using OpenSSL or gosuslugi.ru :

 openssl.exe smime -verify -in C:/dump.xml.sig -content C:/dump.xml -CAfile C:/cartk.pem -inform DER -out C:/validdump.xml 

As a result, we will see the line:

 Verification successful 

and in the validdump.xml file will be the contents of dump.xml .

Now we automate this process. For unloading there is a service working under the SOAP access protocol , the address: the service . WSDL Schema is available at: WSDL Schema .

The service consists of 4 methods:

• getLastDumpDateEx
  The method is designed to get the timestamp of the last update of the download from the registry. In response, it sends the lastDumpDate - a long date comes in the UNIX Timestamp format, but not in seconds, but in milliseconds, as well as the lastDumpDateUrgently - The time when the last change was made to the unloading requiring immediate response , but not in seconds, but in milliseconds).
• getLastDumpDate
  Reserved for compatibility. Similar to getLastDumpDateEx, but returns only one lastDumpDate parameter., A long date comes in the UNIX Timestamp format, not in seconds, but in milliseconds.
• sendRequest
  The method is designed to send a request for receiving an unloading from the registry. Accepts requestFile and signatureFile in base64Binary format — the request file and its signature ( C: /request.xml and C: /request.xml.sign ). boolean format and if all is successful, then code - Request identifier, the string by which you need to get the unloading from the registry in the format string . Also resultComment - Comment to the result of processing the query in the format of string .
• getResult
  The method is designed to get the result of processing the request - unloading from the registry, accepts code - Request identifier, the string by which you need to get the unloading from the registry in string format. In response, it sends result - The result of processing the request in a boolean format, and if everything is successful, then registerZipArchive - A zip-file with downloading from the registry in base64Binary format. Also resultComment - Comment to the result of processing the query in the format of string .

Possible values ​​of the resultComment tag:

• request is being processed
• invalid EP algorithm
• Wrong format of ES
• Invalid ES Certificate
• incorrect value of ES
• certificate verification error
• the applicant does not have a license giving the right to provide services for providing access to the information and telecommunication Internet

Feedback information for resolving issues is provided in the Reminder to the carrier in this section.

The logic of working with the service:

1. Check whether the download is updated from the registry. To do this, call the getLastDumpDateEx method and compare the obtained value with the value obtained at the previous iteration. In case the lastDumpDateUrgently value has changed, then immediately request the updated unloading. In other cases, update the upload at your discretion, but at least once a day .
2. In case the upload is updated, send a request to receive the upload using the sendRequest method.
3. After a few minutes, call the getResult method to get the result of processing the request. If the request has not been processed yet (see the contents of the resultComment field), repeat step 3 in a few minutes.

We add a service to the resources, passing the address of the WSDL scheme.
Function code getLastDumpDate , sendRequest , getResult

 public static Int64 LastDumpDate() { Int64 lastDumpDate = 0; BasicHttpBinding HttpBinding = new BasicHttpBinding(); HttpBinding.MaxReceivedMessageSize = 1*1024*1024*1024; //1Gb using (ChannelFactory<ServiceReference.OperatorRequestPortType> scf = new ChannelFactory<ServiceReference.OperatorRequestPortType>( HttpBinding, new EndpointAddress("http://vigruzki.rkn.gov.ru/services/OperatorRequest/"))) { ServiceReference.OperatorRequestPortType channel = scf.CreateChannel(); ServiceReference.getLastDumpDateResponse glddr = channel.getLastDumpDate(new ServiceReference.getLastDumpDateRequest()); lastDumpDate = glddr.lastDumpDate; } return lastDumpDate; } public static Boolean SendRequest(out String resultComment, out String code, Byte[] requestFile, Byte[] signatureFile) { Boolean result = false; code = null; BasicHttpBinding HttpBinding = new BasicHttpBinding(); HttpBinding.MaxReceivedMessageSize = 1*1024*1024*1024; //1Gb using (ChannelFactory<ServiceReference.OperatorRequestPortType> scf = new ChannelFactory<ServiceReference.OperatorRequestPortType>( HttpBinding, new EndpointAddress("http://vigruzki.rkn.gov.ru/services/OperatorRequest/"))) { ServiceReference.OperatorRequestPortType channel = scf.CreateChannel(); ServiceReference.sendRequestRequestBody srrb = new ServiceReference.sendRequestRequestBody(); srrb.requestFile = requestFile; srrb.signatureFile = signatureFile; ServiceReference.sendRequestResponse srr = channel.sendRequest(new ServiceReference.sendRequestRequest(srrb)); resultComment = srr.Body.resultComment; if (result = srr.Body.result) { code = srr.Body.code; } } return result; } public static Boolean GetResult(out String resultComment, out Byte[] registerZipArchive, String code) { Boolean result = false; registerZipArchive = null; BasicHttpBinding HttpBinding = new BasicHttpBinding(); HttpBinding.MaxReceivedMessageSize = 1*1024*1024*1024; //1Gb using (ChannelFactory<ServiceReference.OperatorRequestPortType> scf = new ChannelFactory<ServiceReference.OperatorRequestPortType>( HttpBinding, new EndpointAddress("http://vigruzki.rkn.gov.ru/services/OperatorRequest/"))) { ServiceReference.OperatorRequestPortType channel = scf.CreateChannel(); ServiceReference.getResultRequestBody grrb = new ServiceReference.getResultRequestBody(); grrb.code = code; ServiceReference.getResultResponse grr = channel.getResult(new ServiceReference.getResultRequest(grrb)); resultComment = grr.Body.resultComment; if (result = grr.Body.result) { registerZipArchive = grr.Body.registerZipArchive; } } return result; } 

For example, send a request:

 String resultComment, code; if(SendRequest(out resultComment, out code, File.ReadAllBytes(@"C:/request.xml"), File.ReadAllBytes(@"C:/request.xml.sign"))) { //...   } 

The timestamp of the last update of the download from the registry can be converted from UNIX Timestamp to DateTime like this:

 DateTime LastDumpDate = (new DateTime(1970, 1, 1, 0, 0, 0, 0)).AddSeconds(LastDumpDate()/1000); 

Parse registry dump

It is necessary to unpack the archive:

 // Byte[] registerZipArchive -     GetResult(); File.WriteAllBytes(@"C:/register.zip", registerZipArchive); ZipFile.ExtractToDirectory(@"C:/register.zip", @"C:/register"); 

As a result, we get a dump of the registry C: /register/dump.xml
Sample dump content:

 <?xml version="1.0" encoding="windows-1251"?> <reg:register updateTime="2014-02-02T12:00:00+04:00" xmlns:reg="http://rsoc.ru" xmlns:tns="http://rsoc.ru" updateTimeUrgently="2014-02-01T11:00:00"> <content id="68" includeTime="2013-12-01T10:00:05"> <decision date="2013-12-01" number="9" org=""/> <url><![CDATA[http://site1.com/index.php]]></url> <domain><![CDATA[site1.com]]></domain> <ip>1.1.1.1</ip> </content> <content id="68" includeTime="2013-12-01T10:00:05"> <decision date="2013-12-01" number="9" org=""/> <url><![CDATA[http://site2.com/page1.php]]></url> <url><![CDATA[http://site2.com/page2.php]]></url> <url><![CDATA[http://site2.com/page3.php]]></url> <domain><![CDATA[site2.com]]></domain> <ip>1.1.1.1</ip> <ip>1.1.1.2</ip> </content> <content id="9999" includeTime="2014-02-01T15:17:51" urgencyType="1"> <decision date="2014-02-01" number=" " org=""/> <url><![CDATA[http://site3.com/page1.html]]></url> <domain><![CDATA[site3.com]]></domain> <ip>1.2.3.4</ip> </content> </reg:register> 

Parsim XML

To do this, create two classes: the first RegisterDump will contain the UpdateTime field and a list of content objects, the Second ItemRegisterDump represents one content object with all fields.

 public class RegisterDump { /* * <reg:register updateTime="2013-07-15T10:05:00+04:00" xmlns:reg="http://rsoc.ru" xmlns:tns="http://rsoc.ru"> * <content></content> * <content></content> * ... * <content></content> * </reg:register> */ public List<ItemRegisterDump> Items { get; set; } public String UpdateTime { get; set; } public RegisterDump() { this.Items = new List<ItemRegisterDump>(); this.UpdateTime = String.Empty; } public RegisterDump(String UpdateTime, List<ItemRegisterDump> Items) { this.Items = Items; this.UpdateTime = UpdateTime; } } public class ItemRegisterDump { /* * <content id="60" includeTime="2013-01-12T16:33:38"> * <decision date="2013-11-03" number="-6" org=""/> * <url><![CDATA[http://habrahabr.ru/post/187574/]]></url> * <ip>123.45.67.89</ip> * </content> * <content id="69" includeTime="2013-05-12T12:43:34"> * <decision date="2013-10-02" number="" org=""/> * <domain><![CDATA[chelaxe.ru]]></domain> * <ip>123.45.67.89</ip> * <ip>87.65.43.210</ip> * </content> */ public String id { get; set; } public String includeTime { get; set; } public String date { get; set; } public String number { get; set; } public String org { get; set; } public List<String> url { get; set; } public List<String> domain { get; set; } public List<String> ip { get; set; } public ItemRegisterDump() { id = String.Empty; includeTime = String.Empty; date = String.Empty; number = String.Empty; org = String.Empty; url = new List<String>(); domain = new List<String>(); ip = new List<String>(); } } 

Parsim:

 RegisterDump Register = new RegisterDump(); String dumpfile = @"C:/register/dump.xml"; XmlDocument xmlDoc = new XmlDocument(); xmlDoc.Load(dumpfile); Register.UpdateTime = xmlDoc.GetElementsByTagName("reg:register")[0].Attributes.GetNamedItem("updateTime").InnerText; XmlNodeList content = xmlDoc.GetElementsByTagName("content"); for (int i = 0; i < content.Count; i++) { ItemRegisterDump item = new ItemRegisterDump(); item.id = content[i].Attributes.GetNamedItem("id").InnerText; item.includeTime = content[i].Attributes.GetNamedItem("includeTime").InnerText; foreach (XmlNode node in content[i].ChildNodes) { switch(node.Name) { case "decision": item.date = node.Attributes.GetNamedItem("date").InnerText; item.number = node.Attributes.GetNamedItem("number").InnerText; item.org = node.Attributes.GetNamedItem("org").InnerText; break; case "url": item.url.Add(node.InnerText); break; case "domain": item.domain.Add(node.InnerText); break; case "ip": item.ip.Add(node.InnerText); break; } } Register.Items.Add(item); } 

Now we block all this stuff on our MikroTik.

Blocked with MikroTik

We do this with the help of a layer7-protocol and add them to the filters .

Here is an example:
/ip firewall layer7-protocol add name=12 comment=register regexp=^.+(chelaxe.ru).*$
/ip firewall filter add action=drop chain=forward disabled=no dst-port=80 layer7-protocol=12 protocol=tcp src-address=192.168.0.0/24 comment=register






All now all packets for the subnet 192.168.0.0/24 coming over TCP on port 80 with the contents of the chelaxe.ru substrings are discarded.
In the comments, the inscription register is not added with a simple. We will need it to delete all the rules before adding the updated ones.

Here is the script that will remove all entries:

/ip firewall layer7-protocol remove [find comment=register]
/ip firewall filter remove [find comment=register]



Communication with MikroTik router will be through the API , the necessary library with examples we have already written: wiki / API C #

Using this library (class) and the principle of content blocking, which we have analyzed above, we implement all this in C #

 public static Boolean AddFilterL7(String ip, String username, String password, RegisterDump dump, String SRCAddress) { Boolean ret = true; try { // MK   http://wiki.mikrotik.com/wiki/API_in_C_Sharp MK mikrotik = new MK(IPAddress.Parse(ip).ToString()); if (mikrotik.Login(username, password)) { mikrotik.Send("/system/script/add"); mikrotik.Send("=name=cleaner"); mikrotik.Send("=source=/ip firewall layer7-protocol remove [find comment=register]\n/ip firewall filter remove [find comment=register]", true); mikrotik.Read(); mikrotik.Send("/system/script/run"); mikrotik.Send("=number=cleaner", true); mikrotik.Read(); /* Cleaner * /ip firewall layer7-protocol remove [find comment=register] * /ip firewall filter remove [find comment=register] */ foreach (ItemRegisterDump item in dump.Items) { for (Int32 i = 0; i < item.domain.Count; i++ ) { mikrotik.Send("/ip/firewall/layer7-protocol/add"); mikrotik.Send("=name=" + item.id + "_" + i); mikrotik.Send("=comment=register"); mikrotik.Send("=regexp=^.+(" + item.domain[i] + ").*$", true); mikrotik.Read(); mikrotik.Send("/ip/firewall/filter/add"); mikrotik.Send("=action=drop"); mikrotik.Send("=chain=forward"); mikrotik.Send("=disabled=no"); mikrotik.Send("=dst-port=80"); mikrotik.Send("=layer7-protocol=" + item.id + "_" + i); mikrotik.Send("=protocol=tcp"); mikrotik.Send("=src-address=" + SRCAddress); mikrotik.Send("=comment=register", true); mikrotik.Read(); } } } } catch (Exception) { ret = false; } return ret; } 

In the example, I used domain locking, but this is not entirely true. For blocking by URL, you can use the following regular expression for layer7-protocol :

^.*(/summary).*(chelaxe.ru).*$

Moreover, the expression should be in lower case when using versions of RouterOS from 5 branches, as a consequence of the error MikroTik .

Sources are available here .

When creating guided:
Memo operator
OpenSSL 1.0.1c + libGOST.so + CryptoPRO + key from ruToken
How we got access to the database of the registry of prohibited resources
Roskomnadzor announced the introduction of the registry of sites with "pirated" movies
RegExr intuitive tool for learning, writing, and testing Regular Expressions