Blocking of forbidden resources of RKN by url including https
Not so long ago, I surfed the forums because of innovations from the RKN, in their “law”, which, besides obscene phrases, does not cause anything to me, but nevertheless we are obliged to comply with these laws. And I discovered that my colleagues use horrible methods to block sites. I decided to share my method, which not only blocks by url, but also knows how to close https.
I’ll say right away that I don’t know whether someone put the similar into the network, I got to everything myself.
What do we need? Do not be afraid of any expensive proprietary solutions from cisco and the like. Generally speaking, I want to say a few kind words to all those who work under the GNU and opensource. Many thanks to you and kowtow. IMHO opensource in front of the entire planet, and I consider it the future. And this article is another confirmation of this.
The idea is as follows:
We create a unique list of blocked IP in ipset. Next, add the rules to NAT PREROUTING. If the destination IP is in the list, then we wrap up on our transparent proxy, where we are filtering by URL.
')
And so from the tools - ipset , iptables , squid (ssl bump).
I did not accidentally sing the praises of opensource, the fact is that today only squid offered a technology that allows you to look into https (at least I don’t know others). And ipset demonstrates the highest speed and keeps thousands of rules without loading the processor and without worsening the network response.
Briefly I will describe the process of receiving the discharge. We download the program P12FromGostCSP, with its help we tear out the private key. Then convert it to pem.
Just in case, check the expiration date
Next, we collect openssl with the support of the Gostov algorithm. How to do this, a lot has been written on the Internet. I will not go into details.
The script itself that receives the upload. (I am not the author of the script and the xml parser, I have only rewritten it for format 2) The script must be called with parameters 0 and 1 at the required interval.
Thus, the result of the script is the presence of 3 files
In general, our task is to get the IP addresses and url in separate files.
We have data for work, now we need to implement a mechanism.
Downloading the current version of squid www.squid-cache.org/Versions
Unpack and compile:
By default, squid is installed in / usr / local / squid
Configuration file, respectively / usr / local / squid / etc
acl - ban accordingly / usr / local / squid / etc / acls
We generate certificates
Next, I give an example of my configuration file:
Now you need to make a list of acls, I use this script
The idea is to add to the end of each url ". *" - this means any sequence of characters. For example, " casino.com . *" Will block the domain and all links including this domain.
You need to perform a few more commands:
On it we will finish with a squid.
Now we need as a "forbidden traffic" to turn on filtering.
To do this, we create two chains in ipset: ZAPRET is a unique list of blocked IP, ZAPRETNET is a list of blocked subnets.
Create the rules themselves using iptables
Now we need to fill in the ipset lists. I quote my script:
That's all, open the browser and try to access the forbidden resource. You should see the squid-a message that access is denied. Something like this:
For the beauty of the plot we replace the files:
On your own page of the type of blocking - "I'm sorry the resource is locked according to the law ....".
PS This blocking method has been tested in combat conditions and still works on real servers. At the same time, the ping did not increase even a millisecond. I didn’t notice much difference in CPU usage either. Since the ILV is very jealous of receiving the unloading, you can uncomment the line in the script:
Replacing of course the phone and mail address on their own. You will receive SMS alert and email alert. For the first, I use the unlocked megaphone Huawei e1550 + gammu modem, for the second I need to configure for example exim4. It is also necessary to enter the details of your organization in the script itself.
PS Since I understand that the good half do not represent what is at all, I decided to give an explanation. A number of laws obliges , I emphasize that each operator is obliged to do this and this is not their whim! And the sanctions for not performing it are very cruel in the flesh before revoking the license! For the banal non-receipt of discharge once a day, a fine of tens of thousands is imposed. And here, for example, the director “presented” him to the person responsible for unloading by installments at half a year.
Here is a list of laws for those interested in who you can read:
I cited only the method that “smoothes out” as much as possible the consequences of these laws. And it allows not to lose access to the host with a blocked url.
Not all your https traffic goes through a proxy, but only the one that goes to a blocked IP . All other https sites you still have access without restrictions.
No one is going to deceive you, you see the left certificate and your right to refuse it and not to visit the monitored resource. For an ordinary user, this is an opportunity to fully use the rest of the unblocked resources of this hoster on a blocked IP. All complaints are welcome here rkn.gov.ru , but from me it is not necessary to do evil of a universal scale.
I’ll say right away that I don’t know whether someone put the similar into the network, I got to everything myself.
What do we need? Do not be afraid of any expensive proprietary solutions from cisco and the like. Generally speaking, I want to say a few kind words to all those who work under the GNU and opensource. Many thanks to you and kowtow. IMHO opensource in front of the entire planet, and I consider it the future. And this article is another confirmation of this.
The idea is as follows:
We create a unique list of blocked IP in ipset. Next, add the rules to NAT PREROUTING. If the destination IP is in the list, then we wrap up on our transparent proxy, where we are filtering by URL.
')
And so from the tools - ipset , iptables , squid (ssl bump).
I did not accidentally sing the praises of opensource, the fact is that today only squid offered a technology that allows you to look into https (at least I don’t know others). And ipset demonstrates the highest speed and keeps thousands of rules without loading the processor and without worsening the network response.
Briefly I will describe the process of receiving the discharge. We download the program P12FromGostCSP, with its help we tear out the private key. Then convert it to pem.
$ / gost-ssl / bin / openssl pkcs12 -in p12.pfx -out provider.pem -nodes -clcerts
Just in case, check the expiration date
$ openssl x509 -in provider.pem -noout -dates
Next, we collect openssl with the support of the Gostov algorithm. How to do this, a lot has been written on the Internet. I will not go into details.
The script itself that receives the upload. (I am not the author of the script and the xml parser, I have only rewritten it for format 2) The script must be called with parameters 0 and 1 at the required interval.
Thus, the result of the script is the presence of 3 files
ip-abuse.txt - IP list
url-abuse.txt - blockable url
subnet-abuse.txt - blocked subnets
In general, our task is to get the IP addresses and url in separate files.
We have data for work, now we need to implement a mechanism.
Downloading the current version of squid www.squid-cache.org/Versions
Unpack and compile:
$. / configure --enable-ssl --enable-ssl-crtd --with-openssl
$ make
$ sudo make install
By default, squid is installed in / usr / local / squid
Configuration file, respectively / usr / local / squid / etc
acl - ban accordingly / usr / local / squid / etc / acls
We generate certificates
openssl req -new -newkey rsa: 1024 -days 3650 -nodes -x509 -keyout myCA.pem -out myCA.pem;
openssl x509 -in myCA.pem -outform DER-out myCA.der
Next, I give an example of my configuration file:
acl deny_url url_regex -i "/ usr / local / squid / etc / acls / zapret" http_access deny deny_url http_access allow all dns_v4_first on http_port 10.20.0.1天128 transparent http_port 3128 #HTTPS https_port 10.20.0.1.40121 transparent ssl-bump generate-host-certificates = on dynamic_cert_mem_cache_size = 4MB cert = / usr / local /squid/etc/myCA.pem sslproxy_flags DONT_VERIFY_PEER sslproxy_cert_error allow all always_direct allow all ssl_bump client-first all ssl_bump server-first all ssl_bump none all sslcrtd_program / usr / local / squid / libexec / ssl_crtd -s / usr / local / squid / var / lib / ssl_db -M 4MB #sslcrtd_children 5 refresh_pattern ^ ftp: 1440 20% 10080 refresh_pattern ^ gopher: 1440 0% 1440 refresh_pattern -i (/ cgi-bin / | \?) 0 0% 0 refresh_pattern. 0 20% 4320
Now you need to make a list of acls, I use this script
#! / bin / bash > / usr / local / squid / etc / acls / zapret cat /gost-ssl21/rzs/dump/url-abuse.txt | sort | uniq | while read LINE; do echo $ LINE '. *' >> / usr / local / squid / etc / acls / zapret done; cat / root / ZAPRET / prokur | sort | uniq >> / usr / local / squid / etc / acls / zapret; / usr / local / squid / bin / squid -k reconfigure exit 0
The idea is to add to the end of each url ". *" - this means any sequence of characters. For example, " casino.com . *" Will block the domain and all links including this domain.
You need to perform a few more commands:
mkdir / usr / local / squid / var / lib
/ usr / local / squid / libexec / ssl_crtd -c -s / usr / local / squid / var / lib / ssl_db
On it we will finish with a squid.
Now we need as a "forbidden traffic" to turn on filtering.
To do this, we create two chains in ipset: ZAPRET is a unique list of blocked IP, ZAPRETNET is a list of blocked subnets.
ipset -N ZAPRETNET hash: net
ipset -N ZAPRET hash: ip
Create the rules themselves using iptables
# Wrap on proxy
iptables -t nat -A PREROUTING -s "subscribers subnet" -p tcp -m set --match-set ZAPRET dst -m tcp --dport 80 -j DNAT --to-destination "proxy IP": 3128
iptables -t nat -A PREROUTING -s "subscribers subnet" -p tcp -m set --match-set ZAPRET dst -m tcp --dport 443 -j DNAT --to-destination "proxy IP": 3129
# Block the entire subnet
iptables -A FORWARD -s "subscribers subnet" -m set - match-set ZAPRETNET dst -j DROP
Now we need to fill in the ipset lists. I quote my script:
#! / bin / bash FILENAME = "create_ruls" > $ FILENAME; echo 'if [-z "` ipset -L | grep ZAPRET` "];' >> $ FILENAME echo 'then' >> $ FILENAME echo 'ipset -N ZAPRET hash: ip' >> $ FILENAME echo 'else' >> $ FILENAME echo 'ipset -F ZAPRET' >> $ FILENAME echo 'fi' >> $ FILENAME echo 'if [-z "` ipset -L | grep ZAPRETNET` "]]' >> $ FILENAME echo 'then' >> $ FILENAME echo 'ipset -N ZAPRETNET hash: net' >> $ FILENAME echo 'else' >> $ FILENAME echo 'ipset -F ZAPRETNET' >> $ FILENAME echo 'fi' >> $ FILENAME cat /gost-ssl21/rzs/dump/ip-abuse.txt | sort | uniq | while read LINE; do echo ipset -A ZAPRET $ LINE >> $ FILENAME done; cat /gost-ssl21/rzs/dump/subnet-abuse.txt | sort | uniq | while read LINE; do echo ipset -A ZAPRETNET $ LINE >> $ FILENAME done;
$ sudo ./create_ruls
That's all, open the browser and try to access the forbidden resource. You should see the squid-a message that access is denied. Something like this:
For the beauty of the plot we replace the files:
/ usr / local / squid / share / errors / en / ERR_ACCESS_DENIED
/ usr / local / squid / share / errors / en / ERR_ACCESS_DENIED
On your own page of the type of blocking - "I'm sorry the resource is locked according to the law ....".
PS This blocking method has been tested in combat conditions and still works on real servers. At the same time, the ping did not increase even a millisecond. I didn’t notice much difference in CPU usage either. Since the ILV is very jealous of receiving the unloading, you can uncomment the line in the script:
system ("/ usr / bin / gammu sendsms TEXT 7910xxxxxxx -len 400 -text 'Get data'; echo 'Unloading from the RKH received' | mail -s' Unloading from the RKH 'received' kopita \ @ mail.ru");
Replacing of course the phone and mail address on their own. You will receive SMS alert and email alert. For the first, I use the unlocked megaphone Huawei e1550 + gammu modem, for the second I need to configure for example exim4. It is also necessary to enter the details of your organization in the script itself.
PS Since I understand that the good half do not represent what is at all, I decided to give an explanation. A number of laws obliges , I emphasize that each operator is obliged to do this and this is not their whim! And the sanctions for not performing it are very cruel in the flesh before revoking the license! For the banal non-receipt of discharge once a day, a fine of tens of thousands is imposed. And here, for example, the director “presented” him to the person responsible for unloading by installments at half a year.
Here is a list of laws for those interested in who you can read:
Federal Law No. 139-FZ of July 28, 2012 amended the following laws of the Russian Federation:
Federal Law of 29 December 2010 No. 436- “On the protection of children from information harmful to their health and development”;
Code of the Russian Federation on administrative offenses;
Federal Law of July 7, 2003 №126- "About communication";
Federal Law of July 27, 2006 №149- “On Information, Information Technologies and Protection of Information”.
I cited only the method that “smoothes out” as much as possible the consequences of these laws. And it allows not to lose access to the host with a blocked url.
Not all your https traffic goes through a proxy, but only the one that goes to a blocked IP . All other https sites you still have access without restrictions.
No one is going to deceive you, you see the left certificate and your right to refuse it and not to visit the monitored resource. For an ordinary user, this is an opportunity to fully use the rest of the unblocked resources of this hoster on a blocked IP. All complaints are welcome here rkn.gov.ru , but from me it is not necessary to do evil of a universal scale.
Source: https://habr.com/ru/post/252933/
All Articles