How did we get access to the database of banned resources

Welcome, habrasoobschestvu.

The most discussed news of the current week in RuNet, of course, is the adoption of Law No. 139-FZ and its consequences. Due to the fact that our company provides hosting services, we needed access to the full registry base in order to promptly respond to the addition of our customers' resources to the database. This is due to the fact that we need to inform customers about the blocking of their resources, because in the case of, for example, blocking the web hosting server's ip, other clients may suffer as well or no fault. Unfortunately, obtaining such access turned out to be not the most obvious, so we would like to share the result of this quest.


')
In the article we would like to focus on the technical aspect and avoid discussing the ethical-political side of the issue, because any active user has already managed to express his dissatisfaction on this topic, regret the root-tracker with Lurkomoryem and think about the topic of circumventing the ban on resources earlier.


Currently, access to a single registry of prohibited resources is assumed to be of two types:

1. For all Internet users in the form of a search on the main page of the registry . Each request requires captcha, in addition, the form, in fact, solves the problem of obtaining information about locking a particular resource known in advance, because if the site is banned by url, then entering into the form a domain name will not give any result. Example: ww.xapka.com is empty, http://www.xapka.com/ - information about blocking.
2. In the form of a full dump of the database of prohibited resources in the section "communication operators" . It is possible to access either through a web form with the need to enter a captcha, or through a SOAP-service. WSDL address: www.zapret-info.gov.ru/services/OperatorRequest/?wsdl .

In the second case, each request for unloading the database must be signed with EDS, in the format PKCS # 7 and the unlinked signature must be sent along with the request. Ideally, we imagined something like this: we buy EDS, take the Debian we are used to, put the appropriate software there to automatically sign requests, write a script that communicates with the zapret-info server via SOAP, generates a request, signs it, downloads a dump and a number of heuristics methods checks to see if any of the resources in the list apply to us.

It would seem that everything is simple, but in fact it turned out to be somewhat more complicated and longer.

EDS

The zapret-info server does not provide any “sandbox” in which it would be possible to test the work of the script without a valid EDS, so the very first thing is to take care of its acquisition.

You can buy a certificate to create a qualified electronic signature in a trusted certification authority . By default, all CAs for signing documents suggest using programs from the CryptoPRO family (CryptoARM, cryptcp). Unfortunately, using this variety of software products overshadows a few facts:

• If the CryptoPRO license is part of the EDS package, then most likely it is intended for Windows.
• CryptoPRO itself does not know how to sign, and tools that can - require a separate license and purchase.
• There are still questions about licensing (whether we need a license for 1 workplace, or for 1 server).

Therefore, a quick search on the Internet brought us to the information that using simple actions with Rutoken, we can teach us how to use OpenSSL, which we are used to. A brief brief of our actions is:

1. Export key in PKCS # 12 format from a cryptocontainer in Windows using the utility P12FromGostCSP
2. Convert it to PEM
3. Register parameters in the openssl.conf config
   At the very beginning, before the sections:
   

   openssl_conf = openssl_def
   

   
   :
   

   [openssl_def]
   engines=engine_section
   
   [engine_section]
   gost=gost_section
   
   [gost_section]
   engine_id=gost
   default_algorithms=ALL
   

4. , :
   

   $ echo "Test" > document.txt
   $ openssl smime -sign -binary -signer ~/sign/gost.crt -inkey ~/sign/gost_nopass.key -outform PEM -in document.txt -out document.txt.sign
   

( «, PKCS#7»). — .

, . , : «! ». :

• XML-, .
• xml- windows-1251.
• XML declaration string (.. , , ).

. SOAP sendRequest(), base64 , . , ( getResult(), ).
, . :

! ( www.zapret-info.gov.ru/tooperators)

: support@rsoc.ru, , . zip-, .

, . :
1. , ;
2. — , , ;
3. ;
4. — , .

:

• SOAP- suds
• lxml.builder xml declaration string, xml - :
  

  from datetime import datetime
  from dateutil.tz import tzlocal
  from lxml import etree
  from lxml.builder import E
  
  request_xml = E.request(
      E.requestTime(datetime.now(tzlocal()).isoformat()),
      E.operatorName(settings.NETANGELS_OPERATOR_NAME),
      E.inn(settings.NETANGELS_INN),
      E.ogrn(settings.NETANGELS_OGRN),
      E.email(settings.NETANGELS_EMAIL),
  )
  #         utf-8 :-(
  request_str = etree.tostring(request_xml, xml_declaration=True, encoding='windows-1251').replace("'", '"')
  

• linux : « ?» Windows :-)
• , .
• «», .
• , , .