As we wrote the URL filtering Rostelecom
Recently, Habré had an article about the introduction of URL filtering by Rostelecom (RTK). It so happened that we offered them to solve this problem a year and a half ago and a year ago they made a solution that passed all the tests and which Rostelecom was ready to include. By the time our shop fell out of mercy, well, yes it is not a question of technology. Therefore, all of the following is the details of our proposed solution. What exactly is implemented, God knows.
Apparently start with DPI. They are in some quantity on the network, but putting them on all traffic would have cost RTC several tens of millions of dollars and would constantly require new investments due to the growth in consumption by the bored population. Neither the previous nor the current leaders have decided on this (perhaps only for now), because the introduction of DPI does not promise any new services with tangible amounts of income.
Since the RTK should have blocked various resources before (the courts decide without regard to the laws), I did it by IP address. Accordingly, the simplest filtering at the boundaries solved the problem. Similarly, they began to solve the problem with the registry of prohibited resources: a specially trained person unloaded the registry, there is both a URL and IP and then added entries to the list, and an access list rules script.
We have automated communication with the registry, and here, as everyone understands, there is no magic (there’s no much of it anymore). Recorded changes in the registry and the implementation of censorship on the network. Added small ryushechki associated with a distributed structure of responsibility on the network of RTK, exclusion from the filter of the part of subscribers (in particular, of the most extreme operators). Well, the courts have not been canceled, so it is possible to make any resource from the court decision. In addition, the management software configures traffic redirection to filtering nodes, prepares reports on user search queries, monitors filtering nodes, and logs all activities.
')
Traffic sent by subscribers to IP addresses corresponding to blocked URLs on PE was turned into a special VPN, where defult looks at a pair of routers in the center. Using SCU / DCU (similar functionality seems to be on pussies), at the border, at the source addresses, traffic from subscribers who are not subject to filtering was excluded. The management software formed / 32 routes according to the IP addresses of the registry, the rules of the config on the central router and the changes took effect. I remember the refusal of sending BGP updates, because it is not good.
Two central routers were configured to copy passing traffic based on the ports. Accordingly, for the entire RTK, several megabits were obtained from strength. It was possible to copy either to an external server or to MS-DPC. And there and there the principle of further processing was the same. The filtering software catches the packet, if in it get with the URL from the list, then in the direction of the server the helmet is reset, and in the direction of the browser redirect to the site with a story about how important moral principles are for modern Russian society.
The minimum response time from the servers included in the registry a year ago exceeded 100 ms, and the software analyzing the URL produced a response 5-6 ms with a load of 3 Gb / s. Therefore, it was decided that to fence a proxy, to ensure its reliability and so on in the same spirit does not make sense. If the filtering software fails, everything will work, except for filtering. True, at the request of RTK, they still designed a couple of filtering servers.
Now it is clear that it was possible and a little easier to do, use a small DPI in the center instead of your filtering software. But we have already done so to another operator. It may be that the selected implementers will combine their registry software with a regular DPI, through which only the selected traffic will go. At the time, we wanted to understand whether there was at least some task that would be worth implementing on the service modules in the router, and not outside. The fact that Juniper has closed access to modules for external developers gives a clear answer to this question, but we were then still tormented by doubts.
As for the cost, of the million in question, more than half were servers, so it was not planned to make a big profit on the software.
DPI and IP blocking
Apparently start with DPI. They are in some quantity on the network, but putting them on all traffic would have cost RTC several tens of millions of dollars and would constantly require new investments due to the growth in consumption by the bored population. Neither the previous nor the current leaders have decided on this (perhaps only for now), because the introduction of DPI does not promise any new services with tangible amounts of income.
Since the RTK should have blocked various resources before (the courts decide without regard to the laws), I did it by IP address. Accordingly, the simplest filtering at the boundaries solved the problem. Similarly, they began to solve the problem with the registry of prohibited resources: a specially trained person unloaded the registry, there is both a URL and IP and then added entries to the list, and an access list rules script.
Filter management software
We have automated communication with the registry, and here, as everyone understands, there is no magic (there’s no much of it anymore). Recorded changes in the registry and the implementation of censorship on the network. Added small ryushechki associated with a distributed structure of responsibility on the network of RTK, exclusion from the filter of the part of subscribers (in particular, of the most extreme operators). Well, the courts have not been canceled, so it is possible to make any resource from the court decision. In addition, the management software configures traffic redirection to filtering nodes, prepares reports on user search queries, monitors filtering nodes, and logs all activities.
')
Traffic rotation and filtering
Traffic sent by subscribers to IP addresses corresponding to blocked URLs on PE was turned into a special VPN, where defult looks at a pair of routers in the center. Using SCU / DCU (similar functionality seems to be on pussies), at the border, at the source addresses, traffic from subscribers who are not subject to filtering was excluded. The management software formed / 32 routes according to the IP addresses of the registry, the rules of the config on the central router and the changes took effect. I remember the refusal of sending BGP updates, because it is not good.
Two central routers were configured to copy passing traffic based on the ports. Accordingly, for the entire RTK, several megabits were obtained from strength. It was possible to copy either to an external server or to MS-DPC. And there and there the principle of further processing was the same. The filtering software catches the packet, if in it get with the URL from the list, then in the direction of the server the helmet is reset, and in the direction of the browser redirect to the site with a story about how important moral principles are for modern Russian society.
The minimum response time from the servers included in the registry a year ago exceeded 100 ms, and the software analyzing the URL produced a response 5-6 ms with a load of 3 Gb / s. Therefore, it was decided that to fence a proxy, to ensure its reliability and so on in the same spirit does not make sense. If the filtering software fails, everything will work, except for filtering. True, at the request of RTK, they still designed a couple of filtering servers.
Now it is clear that it was possible and a little easier to do, use a small DPI in the center instead of your filtering software. But we have already done so to another operator. It may be that the selected implementers will combine their registry software with a regular DPI, through which only the selected traffic will go. At the time, we wanted to understand whether there was at least some task that would be worth implementing on the service modules in the router, and not outside. The fact that Juniper has closed access to modules for external developers gives a clear answer to this question, but we were then still tormented by doubts.
As for the cost, of the million in question, more than half were servers, so it was not planned to make a big profit on the software.
Source: https://habr.com/ru/post/222719/
All Articles