Problems with the list of sites Roskomnadzor. Telecom operators, beware
Hi, Habr.
We have already written how it turned out to be nontrivial to get access to the list of prohibited materials of Roskomnadzor, for organizing automatic monitoring of the access of our networks and sites of our clients for blocking. Unfortunately, the story of the attack on numerous rakes continues, so we would like to share it. The described information will be primarily useful for telecom operators, who, like us, track the situation when resources are on the Roskomnadzor list.
So, having started automatic monitoring in November, we waited for X hours - the appearance of a client resource in the list. On the one hand, of course, it’s pleasant that it’s necessary to rush in contact with the client and decide whether to remove the blocked information, it’s not enough, all the more so as we are told that the reaction takes 1 day, which is written in the instruction sheet to the telecom operator , on the other hand On the other hand, all the tests that were carried out before were somehow synthetic, and the developer inside was waiting impatiently for the automation to work on real data, happily notifying technical support and administrators in all possible ways. After all, it’s nice (for developers, not admins) to see in the logs that the server has crashed, but the backup server has pulled the load on itself, that the master database has fallen off, but the replica has replaced it, etc. In short, it was just interesting to see the system in action.
')
And here, on December 27, we received an e-mail message from Roskomnadzor that the page of our client, www.tourister.ru, got into the ill-fated list. The fact is that Turister.ru is such a social network for tourists, where people post reviews, reviews and life hacks about various trips. And in the blocked article it was told how to go to Amsterdam correctly and what kind of grass worth buying there. It is worth noting that the reason for blocking is completely understandable and does not cause any special questions, so we promptly contacted the administration of the resource and as a result the article was deleted. It would seem that everything was a happy ending, but one moment alerted us - our monitoring of prohibited resources had not responded to this situation in any way. Those. In the staff list, our helpdesk received reports about the next freshly loaded dump, but there was no information that the tourister.ru was blocked:
Of course, such an order of things strongly disturbed us and we began to check the downloaded dumps, because especially in the case of a similar situation, we keep a history of loaded databases.
By the way, about dumps I would like to separately note one interesting feature. As you know, the site zapret-info.gov.ru provides an RPC-interface for unloading the registry database. The documentation declared 3 methods:
In addition, the following case is assumed:
We implemented it, but rather quickly we noticed that each monitoring report contains a freshly downloaded dump, while the file size and the number of documents contained remain unchanged. A quick run of files through the diff-tool showed that the files differ only in the timestamp value. In fact, the date of the last upload is the current time value, with minutes and seconds discarded. As a result, the whole idea of saving traffic and resources just does not work.
So, after reviewing all the dumps for the last few days, we came to the conclusion that they simply keep a record of page blocking on tourister.ru:
At the same time, the zapret-info web interface regularly provides information about blocking:
Of course, we wrote a letter with questions to zapret-info@rsoc.ru , but so far we have not received an answer.
As a conclusion, I would like to quote myself from the previous article on the topic:
So, dear telecom operators, be vigilant.
UPD: In the comments, Deputy Head of Roskomnadzor Ksenzov Ksenzov, Maxim Yuryevich, answers questions related to the work of zapret-info. Attention of users, let's not minus Ksenzov simply because he is a representative of the organization, leading the registry that is hated by everyone, because then we will lose the opportunity to hear the answers and comments from "the other side".
We have already written how it turned out to be nontrivial to get access to the list of prohibited materials of Roskomnadzor, for organizing automatic monitoring of the access of our networks and sites of our clients for blocking. Unfortunately, the story of the attack on numerous rakes continues, so we would like to share it. The described information will be primarily useful for telecom operators, who, like us, track the situation when resources are on the Roskomnadzor list.
So, having started automatic monitoring in November, we waited for X hours - the appearance of a client resource in the list. On the one hand, of course, it’s pleasant that it’s necessary to rush in contact with the client and decide whether to remove the blocked information, it’s not enough, all the more so as we are told that the reaction takes 1 day, which is written in the instruction sheet to the telecom operator , on the other hand On the other hand, all the tests that were carried out before were somehow synthetic, and the developer inside was waiting impatiently for the automation to work on real data, happily notifying technical support and administrators in all possible ways. After all, it’s nice (for developers, not admins) to see in the logs that the server has crashed, but the backup server has pulled the load on itself, that the master database has fallen off, but the replica has replaced it, etc. In short, it was just interesting to see the system in action.
')
And here, on December 27, we received an e-mail message from Roskomnadzor that the page of our client, www.tourister.ru, got into the ill-fated list. The fact is that Turister.ru is such a social network for tourists, where people post reviews, reviews and life hacks about various trips. And in the blocked article it was told how to go to Amsterdam correctly and what kind of grass worth buying there. It is worth noting that the reason for blocking is completely understandable and does not cause any special questions, so we promptly contacted the administration of the resource and as a result the article was deleted. It would seem that everything was a happy ending, but one moment alerted us - our monitoring of prohibited resources had not responded to this situation in any way. Those. In the staff list, our helpdesk received reports about the next freshly loaded dump, but there was no information that the tourister.ru was blocked:
Of course, such an order of things strongly disturbed us and we began to check the downloaded dumps, because especially in the case of a similar situation, we keep a history of loaded databases.
By the way, about dumps I would like to separately note one interesting feature. As you know, the site zapret-info.gov.ru provides an RPC-interface for unloading the registry database. The documentation declared 3 methods:
- getLastDumpDate - returns the timestamp of the last update upload from the registry.
- sendRequest - the method of sending the request, in response to which the result code is returned.
- getResult is a method for getting results.
In addition, the following case is assumed:
- We make a timestamp request, and if it has changed since the last time, we perform further actions. In principle, it is logical - we save traffic and save the server (s?) Of Roskomnadzor from unnecessary load. If the timestamp is unchanged - monitor the resources for this dump.
- We form and send a request. Get the code.
- By code - we get a dump and we check our resources using it.
We implemented it, but rather quickly we noticed that each monitoring report contains a freshly downloaded dump, while the file size and the number of documents contained remain unchanged. A quick run of files through the diff-tool showed that the files differ only in the timestamp value. In fact, the date of the last upload is the current time value, with minutes and seconds discarded. As a result, the whole idea of saving traffic and resources just does not work.
So, after reviewing all the dumps for the last few days, we came to the conclusion that they simply keep a record of page blocking on tourister.ru:
At the same time, the zapret-info web interface regularly provides information about blocking:
Of course, we wrote a letter with questions to zapret-info@rsoc.ru , but so far we have not received an answer.
As a conclusion, I would like to quote myself from the previous article on the topic:
Sometimes a task related to working with government agencies may come to a completely unexpected place.
So, dear telecom operators, be vigilant.
UPD: In the comments, Deputy Head of Roskomnadzor Ksenzov Ksenzov, Maxim Yuryevich, answers questions related to the work of zapret-info. Attention of users, let's not minus Ksenzov simply because he is a representative of the organization, leading the registry that is hated by everyone, because then we will lose the opportunity to hear the answers and comments from "the other side".
Source: https://habr.com/ru/post/164329/
All Articles