Technical aspects of Internet blocking in Russia. Problems and Prospects
Let's start right away with a disclaimer: below there will be basically no political issues. Administrative and legal issues will also be circumvented as much as we can, in order not to completely pull out the technical part from the rest of the planes.
Internet blocking in Russia already exists - this is a reality with which we live and must live on. And if so, you need to understand how it is arranged technically, which the provider can and cannot. Philip Kulin ( schors ) began to collect information on this subject long ago, participated in regulatory work, and went to various meetings. As a result, now only Roskomnadzor knows more about it in Russia, but this is not certain. Under the cut a brief summary of the current state of affairs.
About Speaker: Philip Kulin ( schors ), General Director of Dense Forest LLC - a small Russian hoster, mainly engaged in shared-hosting.
Tangle of problems
It would seem that there are locks and there are. We don’t like them, but maybe there’s nothing wrong with them?
')
Collateral damage is the biggest blocking problem. The most vivid example illustrating this occurred in April 2018, when large blocks of IP addresses of cloud services were blocked, respectively, many services did not work and suffered great damage.
The volatility of regulations and practices that are constantly changing. A year ago, this story would have been completely different, and two years ago it would most likely contradict today's. After a year everything will be different again. Today this is so, in a month - a little bit wrong, and after six months, not at all. It is necessary to follow this, but it is also necessary to have time to work.
Locks difficult to diagnose . If a resource got locked in the registry, this is the simplest case. In the cases that we consider further, it is quite difficult to distinguish a real block from technical problems. A vivid example - in October, Yandex dropped the DNS clock for five, during which time many managed to decide that it was a Roskomnadzor block. It is difficult to determine precisely, indeed, but such situations have already happened, therefore people immediately think about blocking.
It is impossible to predict when you will be blocked and blocked at all. You work quietly, and your work is suddenly over.
It is absolutely impossible to calculate risks , because maybe some widget on the site will fall off, about which you have already forgotten, and maybe the whole business will be hit. A very good example of risk unpredictability is the case of Bitrix24. In March, they very quickly transferred their services to Amazon. In the same month, a document leaked to the network, which truth might have been a fake one, in which large Amazon subnets were registered. Nevertheless, Bitrix24 somehow reacted to it and avoided problems in April, when Amazon services were really blocked.
I assure you that most of you will be so unlucky! Such documents will not by chance flow into your hands. When your business ends, you will know this after the fact.
In simple cases, we know why your site was blocked. For example, the forum posted information that a court recognized as prohibited, but you did not have time to react. But communication with the supervisory authority has unacceptable terms - for example, a day. On the Internet during this time you can lose a fifth of the business.
All this leads to some hopelessness . One can argue with irony over, for example, David Khomak and the blocking of Lurk. But it is quite another thing when it happens to you, as it happened to me once. The client specified the IP addresses of my servers at the domain that I did not manage - I am sitting, and the phone just doesn’t fall silent. Customers say that they are leaving, they demand to return the money, but I can not do anything! And no one can help me with this. This is really a feeling of complete hopelessness.
Risk groups
Locks affect:
How locks work
First, we will briefly discuss how blocking occurs in order to understand the full picture.
It is important to understand that traffic is filtered by each provider. That is, not somewhere on cross-border routers or a state filter, but each provider sets itself a filter between the Internet and subscribers in each of its subnets. In the diagram above, the verification device is next to subscribers, because it pretends to be a subscriber - this is important.
Filtering tools
Providers can buy filtered traffic from a superior provider. But there is a problem - buying traffic from a superior provider, the provider-buyer can not determine the technical problem or blocking. He has no tool, because he receives already cut traffic, and this does not affect his business very well.
Or you can use:
There are no rocket sets there, and the main problems are not about writing a program.
There are the following options for implementation.
For example, you have a small channel of 100 Gbps, you put the filter in the gap.
Some mirror traffic, but with traffic mirroring the problem is that it works as if it is ahead of the curve. That is, the filter tries to respond faster than a normal response, respectively, if the filter began to slow down, - fines (remember, 50-100 thousand rubles).
Selective routing - when traffic to IP addresses, among which there may be something from the “upload”, passes through a separate filter.
Unfortunately, there are no exact figures, but judging by indirect signs and tests, this is now the most common way to filter traffic.
Selective routing can be supplemented by the fact that the sets of IP addresses for filtering are aggregated in a big way. That is, it is not just several addresses that are blocked, but the entire network / 24 falls onto the filter at once. Plus, large providers, for example, in MTS, have special security services that are specifically looking for IP blocks with risk, which also fall into filtering.
Selective routing can also be combined with DNS query filtering . This is a popular method that Dom.ru uses, for example.
Let us examine in stages the problems that all this brings.
Decision on blocking
Roskomnadzor makes a decision - this immediately causes a problem associated, so to speak, with the organization of a support service. In some cases, he must notify the site owner or host, but the addressee of the notification is not accurate (it is taken from public data, but not all support current addresses), notifications are lost, there is no public information .
Because of this, all sorts of bad things happen. The hoster or site owner cannot control what requests are being sent to him, there is no public information. For example, Google has a database of websites with viruses, where you can register yourself as an autonomous system, somehow confirm that you are an autonomous system, and really see for yourself which sites according to Google in your autonomous system distribute "malware". There is no such thing with locks - you expect only that the notification will reach you, and you will have time to read it in time.
The terms of interaction with Roskomnadzor are not respected, and in general, a bit strange, despite the fact that there are standards - to send a notification in 24 hours, to respond in 24 hours, to make a decision in 24 hours. When you receive a notification, and you say that you do not have such information and ask for clarification, you can get an answer in a few weeks. Or maybe you will be blocked, and then they will answer that you still have the information. I had such situations, but very few people sue - I don’t know such cases at all.
Again, if the notification came, you can not always understand what it is about. In most cases, Roskomnadzor normally describes what it means. But even in our practice of a micro-host, there were three cases when the description did not understand what information was involved. I didn’t even know what to write to the client - Roskomnadzor issues protocols and decision texts only by court, although they have such documents.
Time of application "unloading"
So, the decision was made, the provider downloaded the "upload" and then had to do something with it. There are two options for how fast: a day or immediately.
It is important that the day is set aside for locking the resource and unlocking, if suddenly a decision is made to unlock it. For many, this is done as a nightly upgrade of the switch. From my experience: I took the notice at the wrong time, blocked the resource, decided the question, but wait a day until they unblock it. But business does not wait, losses appear.
But now in the regulations very often the word "immediately" sounds , by verbal agreement this is the hour . But there is a verbal agreement today, but not tomorrow. Basically, the phrase “immediately” now concerns prosecutor’s decisions on extremism.
To understand how everything is filtered, you need to know what is inside the “unloading”. There is a list of records in XML of one of four types by types of blocking and details of the solution:
To make it clear what figures are in question, below are the statistics for January 22, 2019.
Important: only 139 thousand records, and the most popular type of blocking is blocking “by domain”. These are HTTP and HTTPS protocols.
Toxicity "unloading"
Before blocking a resource, the provider must parse the "unload". There are problems with this too. I specifically noted that “uploading” is not a registry, but a kind of technical document that is issued to the provider so that it can make decisions based on it. But despite this, the provider has to carry out a very large processing of "unloading".
For example, in the “upload” there is redundancy , the records overlap each other. If you take a URL, it does not mean that there will be no blocking on the domain that is contained in this URL. There are now not so many, though, a little more than three thousand.
The "upload" contains URLs with fragments (#) and sessions . This is generally terrible, because you need to understand how the test goes.
The provider must bring the “upload” to a normal form, because it contains incorrect URLs and domains. Mostly now there are only backslashes. There are gaps, but they are quickly removed, and for some reason, there is a special “love” for the backslash. Well, okay, with backward slashes, the question was decided, and if there is any plus sign? Therefore, there must always be monitoring, there is always something to do.
Once again I pay attention that the problem of the provider is our problem . What does the provider do? A motivation of 100 thousand rubles is a good motivation to make it so that if there is any problem at all, even with any hint of a problem, to chop right away and then figure it out.
The urgency of “unloading” those IP addresses that are not “blocking by IP addresses”, but all others (blocking by domain, by URL, by mask) looks something like this.
Just a little more than half of the time, and the rest is full stuffing.
Check locks
I'll start from the end.
I do not know how abroad, we have absolutely exactly the history of inspections. All locks are not done the way it is written, how to do them, but the way they are checked, because no one likes to pay fines, especially 100 thousand.
Prior to the registry of banned sites, there was a list of extremist materials of the Ministry of Justice (it still exists, it’s just now in the registry, and then it was separate) and prosecutor’s checks of blockings on this list. I am a micro-hoster and I managed to get into blocking my resources.
Existing types of checks:
"Revizor" stands behind the filter and pretends to be a full subscriber. But the device itself does nothing, it receives tasks and gives answers to a certain control center, i.e. This is such a remote shell inside the network provider. It acts very similarly to RIPE Atlas .
The control center of an automated system is a true highload service, because we have 4,000 telecom operators, they do not have one network, but a box must be in each network. That is, not each provider, but in each network of each provider . Accordingly, the control center has certain problems.
Question: Does the check itself have any problems? Of course have.
Verification problems
At SPEKTR-2017 (a forum under the auspices of Roskomnadzor itself), one of the heads of the FSUE of the main radio frequency center (HRDC), A. Veklich made a whole report on what technical problems there are with checking by the “Auditor” of providers.
Verification problems:
The method of work of the AU "Auditor"
In accordance with these problems, the method of work of the Auditor was created.
And yes, your slide exhaustively illustrates this technique. I do not really understand how to live logically with this. For us, this all results in the fact that providers try to solve this problem somehow, and do it in a convenient way for themselves, and not always convenient for us.
Working without a technique, many providers have gained existential experience. In general, the whole method of blocking is an empirical way, which costs money, nerves, falls, even now, when some method has already settled down a bit.
There is an unofficial chat (which is remarkable - in the Telegram, where else), where providers communicate with the staff of the Radio Frequency Center. The most interesting thing is that you can get real help there, HRPC employees help providers solve their problems, and tell how the Auditor works. But this is all unofficial, there are no documents.
There is a Roskomnadzor standard on methods and methods for restricting access, which is registered with the Ministry of Justice. It specifically goes at the end, because it has the lowest priority. Providers do not act as written in the standard, but in the way the method of verification works.
Methods of working with "unloading"
According to the standard and the accumulated experience, I will tell you how they work with “unloading”, that is, how our resources are blocked.
By URL :
Why is there generally a URL with an encrypted protocol is unclear, this is redundancy.
By Domain:
The standard really says that in the second case, blocking “by domain” cannot be blocked by IP address. But the provider, when it begins to knock down the filter, immediately includes another level, so as not to get a penalty. Such a story is not uncommon and naturally leads to additional damage for business.
Theoretically, providers filter by domain with a mask simply as by domain without an asterisk. Since, again, there are no checks, there are no problems either.
By IP address and a block of IP addresses , they are filtered as best they can - right sometimes on the border router.
We know that not only “evil” people are blocked, but also bona fide participants of the Internet . For example, when a person did not intend to keep prohibited information on his hosting, but did not read the letter or did not receive a notification.
The second group is foreign participants. They live in their legal field and do not violate the law. They can laugh, for example, at bribes, they do not see anything wrong with that. For example, hosters do not even have the right to delete this information, because the laws do not apply to them. They are not evil people, but locks hit them.
Let's take a look at the problems, let's talk about DNS filtering, which is recommended by the standard.
First question: where is the DNS ? Indeed, forbidden information can be placed, but DNS represents a service that people need, specifically as DNS, for example, IP addresses. If DNS is fake, everything is not very good, and it is not clear why.
The second point is the implementation of DNS interception . In fact, they intercept all traffic (they simply set up their own caching server - the most common practice), and accordingly the question of quality of service arises. For example, in my office I specifically made the Dom.ru bypass only for DNS, because it is impossible to work when you have to wait for a response from their DNS.
All this can be accelerated by creating your own system of substitution of answers . Then the provider must develop the system and maintain it. They also do this, but this is rare.
When distributing DNS request encryption technology (DNSCrypt, DoT, DoH), it is not known whether the type of DNS blocking will remain.
With domain filtering by mask, the problem is that domains can be on different IPs, so you have to scan the entire HTTP / HTTPS band. But what to do with the other protocols? You are scanning the entire band, but, for example, have banned only the Telegram by mask (by domain) - what to do about it? But there is still no verification !
The next very important point - by the way, this is our future - what to do if on port 443: no SNI, SNI encrypted (ESNI) or other protocols in general, for example, QUIC, DNSCrypt, VPN, MTProto-proxy?
Large companies, such as Google, already support encrypted SNI, while Yandex has DNSCrypt on port 443. Now everyone decides this question in his own way. Some providers, especially mobile ones, if they cannot recognize traffic as HTTPS, simply cut it off. I can’t give exact statistics on this topic, but the approach itself does not inspire optimism.
Domain Resolving
Putting the filter on the entire band, for example, in 100 Gbps is unrealistic. Instead, providers take domains, their IP addresses, and already this traffic is scanned. This is done by large and small providers, mostly large ones, by the way.
In the norm it is not, but "you know!". "Auditor" checks on real addresses and on the fact that in the "unloading", that is, and so and so. Resolving is used because it is profitable for providers. It gives them the opportunity to filter all the same not 100 Gbps, but much less outgoing traffic.
The problem of phases during verification has been known for a long time - everyone is crying, injecting and continuing to do so. The provider took the IP, updated the filters, but before the “Auditor” was checked, the IP changed. The time for which the resource changed IP (the yellow arrow on the diagram) can be measured in milliseconds, but this is enough for the provider to receive a fine.
The second option: after changing the IP resource, the provider and “Auditor” have the same IP, everything was done on time, only the provider updated the filters a millisecond later than “Auditor” checked - with the corresponding damage. It is clear that the provider in such a situation will try to put so many straws that we may not like it.
There are still problems with rezolving:
About a half a year ago, about a thousand domains escaped from blocking. And it is not known, specifically they did it or not. For example, LiveJournal for some reason discards all blocked journals to some CDN, which changes IP by 23 times a minute. Why do so, it is not very clear, but there is such a fact.
Foul Domains
The domain “foul” when it has expired registration, but it remains in the “unloading”, because, of course, no one is following this. The new owner of the domain gains control over the type of blocking that is in the “upload”. In the “upload” there can be subdomains, and when you bought a domain, you get control over all subdomains.
I found about 200 “foul” domains in the “upload”, but in fact there are more of them. We do not know how many domains have already been bought by people who have decided to joke. Maybe they are not, maybe there is.
What does this lead to?
The picture crashes sites Yandex, VKontakte, Wikipedia. On the right is the MSK-IX chart, which MSK-IX itself denies and says it was an internal failure. But for some reason, the internal failure exactly coincided with the DNS attack.
The DNS attack was that someone bought a “rotten” domain, threw thousands of IP addresses there and started watering everything he wants, that is, throwing Yandex, VKontakte, etc. on these IP addresses. Someone played, and so it is still unknown who.
All, of course, puffed out their cheeks and said that this could not be. But despite this, on the graph on the right below, which shows the number of domains in the “unloading”, it is clear that on June 5, 2017 Roskomnadzor carried out a thorough cleaning.
One of the targets of the attack was bank card payments. Banks do not recognize this, because then their shares will fall. But in fact, some banks did undergo DDoS, as a result of which they did not work with payment cards for the whole evening.
Do you think anyone did anything?
March 14, 2018 : another attack vector, but also from the “rotten” domains. Someone bought 4 domains, got control over 400 subdomains, and uploaded 4,000 IP addresses there. TransTeleCom, which, according to Roskomnadzor’s assurances, is all right, somewhere in 600 thousand, the table on the routers overflowed, and 20% of TTK networks throughout the country lay down.
Less than a year:
This is a re-resolution schedule, that is, IP addresses from domains from the “upload”. On May 5, I say to Leonid Evdokimov: “But what if we don’t have a joke - do we not write anything on the chart with a Morse code?” An hour later he did it. By the numbers it is clear that we have calibrated for several thousands of IPs for my rezolving and for my time (I am constantly polling the domains) so that the peaks can be seen and began to write : DIGITAL RESISTANCE.
On the second chart there was a big inscription: “Truly Popov!”, Because it was Radio Day. I cite it, because it shows how Roskomnadzor has finally started cleaning domains. Where there is a step, there they have already found 2 domains, and 2 have not yet, but they brought the mind to some kind of cleansing. At the time of generation, the “rotten” subdomains in the “unloading” inscription were about 4,000, and by November it had become stable, that is, Roskomnadzor was conducting regular cleanings. But, unfortunately, by the time of the article the number of domains has increased again - now there are 1538 of them.
By the way, this is the only result of my activity in a few years - we really started to do at least something!
By the way, about accidents and failures. The fact is that the equipment and filtering programs for some reason do not want to go about the system: somewhere they slow down, failures and mistakes happen . Slightly increases the load, and the quality of service becomes none. You can, for example, use a DNS attack - in the simple case the site will not fall, because everything is normally filtered, but the quality of its work for the user will be very low. You can say so "half the cost" sites to reduce the quality of their work.
For example, last summer at Rostelecom in several regions, the filter for some reason gave out one HTTP protocol chunk incorrectly, and, accordingly, garbage was falling on the screen. The owner of the site Motobratan.ru was on vacation and said that he did not care. The site worked the whole weekend until on Monday morning the engineer came and repaired it.
ISP problems
There should be a good maintenance schedule in case:
But no, it's all fine. There is still a cheatcode here, but I'm afraid it will soon stop working. Providers in such cases, based on their existential experience, turn off the “Auditor”. If you turned off the Auditor, Roskomnadzor calls you, you say: “Oh, it is broken, now we fix it!” But you have a small amount of time between turning off the Inspector and Roskomnadzor’s call, and a little more to fix it. During this time, you can safely carry out some preventive work without fines.
Incidentally, when there was an accident at TTR, many simply turned off the filtering and the “Auditor”, just in case.
Fantasy
Favorite fantasy about what can be done with this is whitelists . But there is but:
The next option is the magic DPI , which the Chinese will sell to us now, and it will work. But miracles do not happen, the performance of the DPI is not at all magical. Secondly, when something complicated is filtered, you need to store states. But DPI is not rubber and, for example, can fall under special attacks on DPI, which will just put it. Normal failures will naturally also. Who lived under the DPI in the closed Internet, knows that they are working so-so.
— DNS , .
: Telegram?
, : . Telegram , , , DPI . , , , Telegram.
, .
, , Telegram- @usher2 schors
Internet blocking in Russia already exists - this is a reality with which we live and must live on. And if so, you need to understand how it is arranged technically, which the provider can and cannot. Philip Kulin ( schors ) began to collect information on this subject long ago, participated in regulatory work, and went to various meetings. As a result, now only Roskomnadzor knows more about it in Russia, but this is not certain. Under the cut a brief summary of the current state of affairs.
About Speaker: Philip Kulin ( schors ), General Director of Dense Forest LLC - a small Russian hoster, mainly engaged in shared-hosting.
Tangle of problems
It would seem that there are locks and there are. We don’t like them, but maybe there’s nothing wrong with them?
')
In fact, locking is a solid tangle of problems.
Collateral damage is the biggest blocking problem. The most vivid example illustrating this occurred in April 2018, when large blocks of IP addresses of cloud services were blocked, respectively, many services did not work and suffered great damage.
The volatility of regulations and practices that are constantly changing. A year ago, this story would have been completely different, and two years ago it would most likely contradict today's. After a year everything will be different again. Today this is so, in a month - a little bit wrong, and after six months, not at all. It is necessary to follow this, but it is also necessary to have time to work.
Locks difficult to diagnose . If a resource got locked in the registry, this is the simplest case. In the cases that we consider further, it is quite difficult to distinguish a real block from technical problems. A vivid example - in October, Yandex dropped the DNS clock for five, during which time many managed to decide that it was a Roskomnadzor block. It is difficult to determine precisely, indeed, but such situations have already happened, therefore people immediately think about blocking.
It is impossible to predict when you will be blocked and blocked at all. You work quietly, and your work is suddenly over.
It is absolutely impossible to calculate risks , because maybe some widget on the site will fall off, about which you have already forgotten, and maybe the whole business will be hit. A very good example of risk unpredictability is the case of Bitrix24. In March, they very quickly transferred their services to Amazon. In the same month, a document leaked to the network, which truth might have been a fake one, in which large Amazon subnets were registered. Nevertheless, Bitrix24 somehow reacted to it and avoided problems in April, when Amazon services were really blocked.
I assure you that most of you will be so unlucky! Such documents will not by chance flow into your hands. When your business ends, you will know this after the fact.
In simple cases, we know why your site was blocked. For example, the forum posted information that a court recognized as prohibited, but you did not have time to react. But communication with the supervisory authority has unacceptable terms - for example, a day. On the Internet during this time you can lose a fifth of the business.
All this leads to some hopelessness . One can argue with irony over, for example, David Khomak and the blocking of Lurk. But it is quite another thing when it happens to you, as it happened to me once. The client specified the IP addresses of my servers at the domain that I did not manage - I am sitting, and the phone just doesn’t fall silent. Customers say that they are leaving, they demand to return the money, but I can not do anything! And no one can help me with this. This is really a feeling of complete hopelessness.
Risk groups
Locks affect:
- Owners of sites and services , which, including, may block by mistake. Or they may recognize some information as prohibited.
- Users of the lock service also concern. The fact that your site is blocked, concerns you in the first place. But there are people who used this site or service.
- Hosters and providers that are between two lights. Because they are required and running services, and the requirements of the supervisory authority, which faces fines. Let me remind you that fines from 50 to 100 thousand rubles for the protocol. For example, such protocols for a month can be many and the amounts are very substantial.
How locks work
First, we will briefly discuss how blocking occurs in order to understand the full picture.
- Federal executive authorities or the court decide to prohibit any information for some reason.
- Send information to Roskomnadzor, which must make an entry in the registry of prohibited sites.
- Then there are some internal procedures (there are too many of them - you can read a whole lecture), as a result of which Roskomnadzor may decide to block and enter the site into the so-called “upload” - a technical file that is sent to providers.
- Providers on this file implement the restriction.
- Check providers, which has been happening for two years automatically.
It is important to understand that traffic is filtered by each provider. That is, not somewhere on cross-border routers or a state filter, but each provider sets itself a filter between the Internet and subscribers in each of its subnets. In the diagram above, the verification device is next to subscribers, because it pretends to be a subscriber - this is important.
Filtering tools
Providers can buy filtered traffic from a superior provider. But there is a problem - buying traffic from a superior provider, the provider-buyer can not determine the technical problem or blocking. He has no tool, because he receives already cut traffic, and this does not affect his business very well.
Or you can use:
- special complex commercial solutions;
- free open source solutions (currently there is only one such project);
- your "collective farm".
There are no rocket sets there, and the main problems are not about writing a program.
There are the following options for implementation.
For example, you have a small channel of 100 Gbps, you put the filter in the gap.
Some mirror traffic, but with traffic mirroring the problem is that it works as if it is ahead of the curve. That is, the filter tries to respond faster than a normal response, respectively, if the filter began to slow down, - fines (remember, 50-100 thousand rubles).
Selective routing - when traffic to IP addresses, among which there may be something from the “upload”, passes through a separate filter.
Unfortunately, there are no exact figures, but judging by indirect signs and tests, this is now the most common way to filter traffic.
Selective routing can be supplemented by the fact that the sets of IP addresses for filtering are aggregated in a big way. That is, it is not just several addresses that are blocked, but the entire network / 24 falls onto the filter at once. Plus, large providers, for example, in MTS, have special security services that are specifically looking for IP blocks with risk, which also fall into filtering.
Selective routing can also be combined with DNS query filtering . This is a popular method that Dom.ru uses, for example.
Let us examine in stages the problems that all this brings.
Decision on blocking
Roskomnadzor makes a decision - this immediately causes a problem associated, so to speak, with the organization of a support service. In some cases, he must notify the site owner or host, but the addressee of the notification is not accurate (it is taken from public data, but not all support current addresses), notifications are lost, there is no public information .
Because of this, all sorts of bad things happen. The hoster or site owner cannot control what requests are being sent to him, there is no public information. For example, Google has a database of websites with viruses, where you can register yourself as an autonomous system, somehow confirm that you are an autonomous system, and really see for yourself which sites according to Google in your autonomous system distribute "malware". There is no such thing with locks - you expect only that the notification will reach you, and you will have time to read it in time.
The terms of interaction with Roskomnadzor are not respected, and in general, a bit strange, despite the fact that there are standards - to send a notification in 24 hours, to respond in 24 hours, to make a decision in 24 hours. When you receive a notification, and you say that you do not have such information and ask for clarification, you can get an answer in a few weeks. Or maybe you will be blocked, and then they will answer that you still have the information. I had such situations, but very few people sue - I don’t know such cases at all.
Again, if the notification came, you can not always understand what it is about. In most cases, Roskomnadzor normally describes what it means. But even in our practice of a micro-host, there were three cases when the description did not understand what information was involved. I didn’t even know what to write to the client - Roskomnadzor issues protocols and decision texts only by court, although they have such documents.
Time of application "unloading"
So, the decision was made, the provider downloaded the "upload" and then had to do something with it. There are two options for how fast: a day or immediately.
It is important that the day is set aside for locking the resource and unlocking, if suddenly a decision is made to unlock it. For many, this is done as a nightly upgrade of the switch. From my experience: I took the notice at the wrong time, blocked the resource, decided the question, but wait a day until they unblock it. But business does not wait, losses appear.
But now in the regulations very often the word "immediately" sounds , by verbal agreement this is the hour . But there is a verbal agreement today, but not tomorrow. Basically, the phrase “immediately” now concerns prosecutor’s decisions on extremism.
To understand how everything is filtered, you need to know what is inside the “unloading”. There is a list of records in XML of one of four types by types of blocking and details of the solution:
- URL (s) + domain + IP address (a);
- Domain + IP Address (s);
- Domain with mask (* .example.com) + IP-address (a);
- IP-address (a) - it is clear, there is an address, and we block it completely.
To make it clear what figures are in question, below are the statistics for January 22, 2019.
Important: only 139 thousand records, and the most popular type of blocking is blocking “by domain”. These are HTTP and HTTPS protocols.
Toxicity "unloading"
Before blocking a resource, the provider must parse the "unload". There are problems with this too. I specifically noted that “uploading” is not a registry, but a kind of technical document that is issued to the provider so that it can make decisions based on it. But despite this, the provider has to carry out a very large processing of "unloading".
For example, in the “upload” there is redundancy , the records overlap each other. If you take a URL, it does not mean that there will be no blocking on the domain that is contained in this URL. There are now not so many, though, a little more than three thousand.
The "upload" contains URLs with fragments (#) and sessions . This is generally terrible, because you need to understand how the test goes.
The provider must bring the “upload” to a normal form, because it contains incorrect URLs and domains. Mostly now there are only backslashes. There are gaps, but they are quickly removed, and for some reason, there is a special “love” for the backslash. Well, okay, with backward slashes, the question was decided, and if there is any plus sign? Therefore, there must always be monitoring, there is always something to do.
Once again I pay attention that the problem of the provider is our problem . What does the provider do? A motivation of 100 thousand rubles is a good motivation to make it so that if there is any problem at all, even with any hint of a problem, to chop right away and then figure it out.
The urgency of “unloading” those IP addresses that are not “blocking by IP addresses”, but all others (blocking by domain, by URL, by mask) looks something like this.
Just a little more than half of the time, and the rest is full stuffing.
Check locks
I'll start from the end.
The whole history of the implementation of locks in Russia is the history of checks on these locks.
I do not know how abroad, we have absolutely exactly the history of inspections. All locks are not done the way it is written, how to do them, but the way they are checked, because no one likes to pay fines, especially 100 thousand.
Prior to the registry of banned sites, there was a list of extremist materials of the Ministry of Justice (it still exists, it’s just now in the registry, and then it was separate) and prosecutor’s checks of blockings on this list. I am a micro-hoster and I managed to get into blocking my resources.
Existing types of checks:
- Field inspections (mainly the Ministry of Internal Affairs, the FSB and the prosecutor's office) are rare, but there are some.
- AS “Revizor” is a favorite automated system that checks all providers.
"Revizor" stands behind the filter and pretends to be a full subscriber. But the device itself does nothing, it receives tasks and gives answers to a certain control center, i.e. This is such a remote shell inside the network provider. It acts very similarly to RIPE Atlas .
The control center of an automated system is a true highload service, because we have 4,000 telecom operators, they do not have one network, but a box must be in each network. That is, not each provider, but in each network of each provider . Accordingly, the control center has certain problems.
Question: Does the check itself have any problems? Of course have.
Verification problems
At SPEKTR-2017 (a forum under the auspices of Roskomnadzor itself), one of the heads of the FSUE of the main radio frequency center (HRDC), A. Veklich made a whole report on what technical problems there are with checking by the “Auditor” of providers.
Verification problems:
- Does “Revizor” see this as a provider or resource filter? Maybe it was someone of you who found the database of the "Auditor" and all the sites just give a stub, similar to the provider stub.
- Block rate - for all types of blocking (for HTTPS, domain, IP-address), what is the indication that the resource is locked? For example, by the IP address you need to scan the ports, this is a whole problem.
- Blocking rate for other protocols.
- How to check domain mask ? Under the asterisk can be different IP-addresses, different domains. Can I put a filter on the whole strip? And to check what - selectively or some kind of hash to generate? I’ll say right away that there is a regulatory blocking, but no one checks it.
The method of work of the AU "Auditor"
In accordance with these problems, the method of work of the Auditor was created.
And yes, your slide exhaustively illustrates this technique. I do not really understand how to live logically with this. For us, this all results in the fact that providers try to solve this problem somehow, and do it in a convenient way for themselves, and not always convenient for us.
Working without a technique, many providers have gained existential experience. In general, the whole method of blocking is an empirical way, which costs money, nerves, falls, even now, when some method has already settled down a bit.
There is an unofficial chat (which is remarkable - in the Telegram, where else), where providers communicate with the staff of the Radio Frequency Center. The most interesting thing is that you can get real help there, HRPC employees help providers solve their problems, and tell how the Auditor works. But this is all unofficial, there are no documents.
There is a Roskomnadzor standard on methods and methods for restricting access, which is registered with the Ministry of Justice. It specifically goes at the end, because it has the lowest priority. Providers do not act as written in the standard, but in the way the method of verification works.
Methods of working with "unloading"
According to the standard and the accumulated experience, I will tell you how they work with “unloading”, that is, how our resources are blocked.
By URL :
- If this is the usual HTTP protocol filter by headers.
- If the encrypted protocol is HTTPS - filter as "domain"
Why is there generally a URL with an encrypted protocol is unclear, this is redundancy.
By Domain:
- The usual HTTP protocol is a filter by the Host header.
- Encrypted HTTPS protocol - or DNS filter; or filter by SNI header (when an encrypted connection is established, an unencrypted header with a domain name is transmitted inside); or filter by IP address.
- The rest of the protocols are recommended to be blocked as by IP-address, but since the “Auditor” does not check this, someone does it, someone does not.
The standard really says that in the second case, blocking “by domain” cannot be blocked by IP address. But the provider, when it begins to knock down the filter, immediately includes another level, so as not to get a penalty. Such a story is not uncommon and naturally leads to additional damage for business.
Theoretically, providers filter by domain with a mask simply as by domain without an asterisk. Since, again, there are no checks, there are no problems either.
By IP address and a block of IP addresses , they are filtered as best they can - right sometimes on the border router.
Bona fide members
We know that not only “evil” people are blocked, but also bona fide participants of the Internet . For example, when a person did not intend to keep prohibited information on his hosting, but did not read the letter or did not receive a notification.
The second group is foreign participants. They live in their legal field and do not violate the law. They can laugh, for example, at bribes, they do not see anything wrong with that. For example, hosters do not even have the right to delete this information, because the laws do not apply to them. They are not evil people, but locks hit them.
Filtering problems
Let's take a look at the problems, let's talk about DNS filtering, which is recommended by the standard.
First question: where is the DNS ? Indeed, forbidden information can be placed, but DNS represents a service that people need, specifically as DNS, for example, IP addresses. If DNS is fake, everything is not very good, and it is not clear why.
The second point is the implementation of DNS interception . In fact, they intercept all traffic (they simply set up their own caching server - the most common practice), and accordingly the question of quality of service arises. For example, in my office I specifically made the Dom.ru bypass only for DNS, because it is impossible to work when you have to wait for a response from their DNS.
All this can be accelerated by creating your own system of substitution of answers . Then the provider must develop the system and maintain it. They also do this, but this is rare.
When distributing DNS request encryption technology (DNSCrypt, DoT, DoH), it is not known whether the type of DNS blocking will remain.
With domain filtering by mask, the problem is that domains can be on different IPs, so you have to scan the entire HTTP / HTTPS band. But what to do with the other protocols? You are scanning the entire band, but, for example, have banned only the Telegram by mask (by domain) - what to do about it? But there is still no verification !
The next very important point - by the way, this is our future - what to do if on port 443: no SNI, SNI encrypted (ESNI) or other protocols in general, for example, QUIC, DNSCrypt, VPN, MTProto-proxy?
Large companies, such as Google, already support encrypted SNI, while Yandex has DNSCrypt on port 443. Now everyone decides this question in his own way. Some providers, especially mobile ones, if they cannot recognize traffic as HTTPS, simply cut it off. I can’t give exact statistics on this topic, but the approach itself does not inspire optimism.
Domain Resolving
Putting the filter on the entire band, for example, in 100 Gbps is unrealistic. Instead, providers take domains, their IP addresses, and already this traffic is scanned. This is done by large and small providers, mostly large ones, by the way.
In the norm it is not, but "you know!". "Auditor" checks on real addresses and on the fact that in the "unloading", that is, and so and so. Resolving is used because it is profitable for providers. It gives them the opportunity to filter all the same not 100 Gbps, but much less outgoing traffic.
The problem of phases during verification has been known for a long time - everyone is crying, injecting and continuing to do so. The provider took the IP, updated the filters, but before the “Auditor” was checked, the IP changed. The time for which the resource changed IP (the yellow arrow on the diagram) can be measured in milliseconds, but this is enough for the provider to receive a fine.
The second option: after changing the IP resource, the provider and “Auditor” have the same IP, everything was done on time, only the provider updated the filters a millisecond later than “Auditor” checked - with the corresponding damage. It is clear that the provider in such a situation will try to put so many straws that we may not like it.
There are still problems with rezolving:
- balancing;
- geo-targeting;
- service migration, including intentional.
About a half a year ago, about a thousand domains escaped from blocking. And it is not known, specifically they did it or not. For example, LiveJournal for some reason discards all blocked journals to some CDN, which changes IP by 23 times a minute. Why do so, it is not very clear, but there is such a fact.
Foul Domains
The domain “foul” when it has expired registration, but it remains in the “unloading”, because, of course, no one is following this. The new owner of the domain gains control over the type of blocking that is in the “upload”. In the “upload” there can be subdomains, and when you bought a domain, you get control over all subdomains.
I found about 200 “foul” domains in the “upload”, but in fact there are more of them. We do not know how many domains have already been bought by people who have decided to joke. Maybe they are not, maybe there is.
What does this lead to?
The picture crashes sites Yandex, VKontakte, Wikipedia. On the right is the MSK-IX chart, which MSK-IX itself denies and says it was an internal failure. But for some reason, the internal failure exactly coincided with the DNS attack.
The DNS attack was that someone bought a “rotten” domain, threw thousands of IP addresses there and started watering everything he wants, that is, throwing Yandex, VKontakte, etc. on these IP addresses. Someone played, and so it is still unknown who.
All, of course, puffed out their cheeks and said that this could not be. But despite this, on the graph on the right below, which shows the number of domains in the “unloading”, it is clear that on June 5, 2017 Roskomnadzor carried out a thorough cleaning.
One of the targets of the attack was bank card payments. Banks do not recognize this, because then their shares will fall. But in fact, some banks did undergo DDoS, as a result of which they did not work with payment cards for the whole evening.
Do you think anyone did anything?
March 14, 2018 : another attack vector, but also from the “rotten” domains. Someone bought 4 domains, got control over 400 subdomains, and uploaded 4,000 IP addresses there. TransTeleCom, which, according to Roskomnadzor’s assurances, is all right, somewhere in 600 thousand, the table on the routers overflowed, and 20% of TTK networks throughout the country lay down.
Less than a year:
This is a re-resolution schedule, that is, IP addresses from domains from the “upload”. On May 5, I say to Leonid Evdokimov: “But what if we don’t have a joke - do we not write anything on the chart with a Morse code?” An hour later he did it. By the numbers it is clear that we have calibrated for several thousands of IPs for my rezolving and for my time (I am constantly polling the domains) so that the peaks can be seen and began to write : DIGITAL RESISTANCE.
On the second chart there was a big inscription: “Truly Popov!”, Because it was Radio Day. I cite it, because it shows how Roskomnadzor has finally started cleaning domains. Where there is a step, there they have already found 2 domains, and 2 have not yet, but they brought the mind to some kind of cleansing. At the time of generation, the “rotten” subdomains in the “unloading” inscription were about 4,000, and by November it had become stable, that is, Roskomnadzor was conducting regular cleanings. But, unfortunately, by the time of the article the number of domains has increased again - now there are 1538 of them.
By the way, this is the only result of my activity in a few years - we really started to do at least something!
By the way, about accidents and failures. The fact is that the equipment and filtering programs for some reason do not want to go about the system: somewhere they slow down, failures and mistakes happen . Slightly increases the load, and the quality of service becomes none. You can, for example, use a DNS attack - in the simple case the site will not fall, because everything is normally filtered, but the quality of its work for the user will be very low. You can say so "half the cost" sites to reduce the quality of their work.
For example, last summer at Rostelecom in several regions, the filter for some reason gave out one HTTP protocol chunk incorrectly, and, accordingly, garbage was falling on the screen. The owner of the site Motobratan.ru was on vacation and said that he did not care. The site worked the whole weekend until on Monday morning the engineer came and repaired it.
ISP problems
There should be a good maintenance schedule in case:
- the unloading service does not work;
- an accident has occurred on the communication channels, “unloading” is impossible to obtain;
- filters broke;
- filters on the prevention.
But no, it's all fine. There is still a cheatcode here, but I'm afraid it will soon stop working. Providers in such cases, based on their existential experience, turn off the “Auditor”. If you turned off the Auditor, Roskomnadzor calls you, you say: “Oh, it is broken, now we fix it!” But you have a small amount of time between turning off the Inspector and Roskomnadzor’s call, and a little more to fix it. During this time, you can safely carry out some preventive work without fines.
Incidentally, when there was an accident at TTR, many simply turned off the filtering and the “Auditor”, just in case.
Fantasy
Favorite fantasy about what can be done with this is whitelists . But there is but:
- Whitelisting criteria.
- Interaction system - the support of Roskomnadzor is not very good, one can imagine what it will become.
- In the clouds, how to work - write a request for each virtual? For example, Kubernetes wants to create a node, and you write an application for adding it to the white list - hello Continuous Integration!
The next option is the magic DPI , which the Chinese will sell to us now, and it will work. But miracles do not happen, the performance of the DPI is not at all magical. Secondly, when something complicated is filtered, you need to store states. But DPI is not rubber and, for example, can fall under special attacks on DPI, which will just put it. Normal failures will naturally also. Who lived under the DPI in the closed Internet, knows that they are working so-so.
— DNS , .
- , DNS , — .
- , . , , .
- , DNS, «» .
: Telegram?
, : . Telegram , , , DPI . , , , Telegram.
, .
, , Telegram- @usher2 schors
HighLoad++ 2018. — , . , , Saint HighLoad++ .
Source: https://habr.com/ru/post/431188/
All Articles