Transparent HTTPS proxy to bypass Roskomnadzor’s locks
Hi, habrovchane! I think many have recently encountered problems of access to the necessary resources because of the shame of Roskom’s supervision of blocking Telegrams. And I think there are no comments here. The fact is that these resources are not to blame for anything, but they are blocked. Problems arose with Viber, ReCaptcha, GoogleFonts, Youtube and others (except the telegraph itself). It happened in my organization, and we need some innocent services like air. At some time, everything was decided by using proxy servers, but they were unstable or completely disconnected (they were also blocked by our great and mighty RKN).
After reading a bunch of articles, I got the idea to teach Squid to start up separate URLs via Tor. Whether to use this method, you decide. But I will say that after the implementation, all the problems that existed before have disappeared. Who cares, go under the cat.
So first, a little theory. As we all know, Tor is not an HTTP proxy, it cannot be made a direct peer for the underlying Squid. It provides SOCKS proxying (of course, not only, but we need just that). To marry Tor from Squid, we need something that could play the role of a conductor from Tor to Squid and back. And of course, ladies and gentlemen, this is Privoxy. Just the same, he is able to be a direct peer, and send everything further to Tor.
')
It was, as I said, read a bunch of articles, but none suited me. I came across this article, but it didn’t quite fit me either, since I don’t need a bump. In general, all available articles, almost all, imply either a bump, or just http, and in my case, you need HTTPS, splice, and transparency. Also seen this and this , but there are very different approaches. Its pros and cons. I chose for myself a bunch of Squid + Tor.
I already wrote about how to make Squid transparent with HTTPS proxying without certificate substitution. And of course, I tried to implement the idea on it. But I was disappointed. HTTP requests were fine with TOR, but no HTTPS. The problem is not very well known, and I learned from one of the developers that this is a flaw in older versions of Squid. But during the experiments, a solution was found - Squid 3.5.27, which fixed this bug + beautiful domain names in the logs (https), instead of ip addresses. But even here I was waited by several disappointments, which will be discussed below. But everything, as they say, is finished with a file.
So, the source data:
Build Squid with pens or install ready-made packages (links below), you decide. If you think that I’ve seenblackjack and w… viruses in them, then compile yourself. You also need to compile if you have a different distro (if Ubunt, put it). We will collect the version of Squid 3.5.23, which at the time of writing this article is lying in the Stretch repository, upgrading it to the latest source code 3.5.27 from the offsite site. Unlike my first article on HTTPS + Squid, we will be compiling without libressl.
So prepare for the build:
Next we get the source code for Squid 3.5.23
We download this particular archive with Squid sources:
Go to the Squid source directory and update the source code to the newly added sorts:
Go to the new directory with updated sources:
Add to debian / rules options for compiling:
Next, you need to patch the source code with this patch:
What is it for? I will explain. When I wrote the first article about peek and splice, I said that newer versions do not work, and it was like that, and this patch corrects the very problem that Squid selectively tears off HTTPS connections with an interesting By posting to cache.log:
The fact is that on one host something resolves in one IP, on a neighboring one sometimes in another, on Squid itself in the third, since There is a DNS cache and it is not updated synchronously. Squid does not match the ip-domain in its cache (because it updated its cache a little earlier or later) and terminates the connection. It seems like protection, but nowadays this is considered normal (round-robin DNS). Developers reinsured. And we do not need it at all! To those who say that this patch probably carries a security risk, I will answer that I have consulted on this patch with Yuri Voinov, who is directly related to the Squid development team. There is no threat here!
So, the file for the patch was created, the code was dropped, you need to patch:
Next, you need to cancel the application of one patch when compiling (otherwise you will receive an error that this patch cannot be applied, since it has already been applied). Go to debian / patches / series and comment out there 0003-SQUID-2018_1.patch , putting a # sign in front of it:
Well, then - compile and build packages!
Install squid-langpack
and install fresh packages
If apt curses dependency, do
Next, you need to turn off Squid from autoload (by default, the init file is used, Squid complains that the PID file is unavailable)
and create a systemd service in the / etc / systemd / system directory (the service file is in the source code, and completely copied here)
Turn it on
Install tor, privoxy
I personally did not touch the Tor configuration at all, but the Privoxy config can be brought to this form:
Almost done. Go to the directory / etc / squid , something we change there. Create a pem file for splice:
And we will bring squid.conf to the following form:
The url_regex list looks like this (the list is given for an example!):
For more information about this format of the list, read in the documentation. The result did not make me wait long - it all worked as planned. And it works to this day. Perhaps the article was late, but it will probably come in handy in the future).
Topic will replenish whenever possible.
Ready Squid Packages (.deb x86)
UPD 04.05.18:you can add a line to torcc
and, in principle, abandon Privoxy. Thank you dartraiden for noting
Found a bug. Using HTTPTunnelPort is not recommended, Privoxy is needed until — until this is closed. Thanks to Yuri Voinov!
Also corrected the article on the part of libecap3. You do not need to compile anything, but simply install it from the Stretch repository. Thanks AlucoST for the comment
UPD 05/05/18: Comrade dartraiden suggests that you can add to the Tor config
This excludes the use of output nodes in specified countries.
After reading a bunch of articles, I got the idea to teach Squid to start up separate URLs via Tor. Whether to use this method, you decide. But I will say that after the implementation, all the problems that existed before have disappeared. Who cares, go under the cat.
Why is this?
The article is written solely for the purpose of helping those who wrongfully suffer from the total nonsense that is happening in our country. It is also aimed at those who need exactly “transparent” Squid with HTTPS without certificate substitution and the ability to track visits both via HTTP and HTTPS, since the article is essentially a continuation of another article of mine , because here I propose a fix for a long-standing bug that did not allow to see https resources in the logs and did not allow using newer versions of Squid. Well, and also just for those who are interested in using Squid.
What are the benefits?
- Unlimited scaling options
- Relative ease of support and administration
- If this is important, then you can provide anonymous access to the resources listed in the list (although this is not the topic of the article, anonymity is provided by the “mess”
- Stability. If you use several Tor services (different config files), you can connect them to Squid, and get a round robin.
- Absolute free. Forever and ever.
I don't know anything, Squid \ Tor is slow, I will go and pick up a VPS with VPN over the hill
Congratulations! You really decided that Roskomnadzor caused you inconvenience, and you still have to pay for it to get out of the situation? OK. Feel free to skip the article, and raise VPN tunnels. By the way, VPN is successfully blocked. Easy. And in the light of recent events, I will tell you that soon no one will be able to use VPS to bypass locks at the level of the law. And plus to everything, your VPS can also be locked into the lock "simply because there is a telegram sitting next to it." Tor won't block, no way. If you configure it with obfs, then never anywhere (subject, perhaps, for a separate article, since this obfs is not considered). And how much will you need to raise such VPS with VPN? How to serve it? Here, the solution is easier at times, reliable, and, if desired, quite speedy, and free. So, before you mislead other readers, please think again about everything + and - VPS with VPN, and only then state that Squid + Tor is a slow, unreliable solution.
Tor will block! In China, there is already blocked
Not. Not. And again, no. In China, Tor works with a configured obfs. It works great. There is no way to block Tor in the World. Even China with its capacity, minds and finances could not do it.
Tor slow! And if you work through obfs, then generally horror!
Refer to the of.documentation and a bunch of articles on the Internet, which describes how to make the speed was at a decent level. And again, by setting up several copies of Tor with different configs, you can attach them to Squid and get a round robin.
So first, a little theory. As we all know, Tor is not an HTTP proxy, it cannot be made a direct peer for the underlying Squid. It provides SOCKS proxying (of course, not only, but we need just that). To marry Tor from Squid, we need something that could play the role of a conductor from Tor to Squid and back. And of course, ladies and gentlemen, this is Privoxy. Just the same, he is able to be a direct peer, and send everything further to Tor.
')
It was, as I said, read a bunch of articles, but none suited me. I came across this article, but it didn’t quite fit me either, since I don’t need a bump. In general, all available articles, almost all, imply either a bump, or just http, and in my case, you need HTTPS, splice, and transparency. Also seen this and this , but there are very different approaches. Its pros and cons. I chose for myself a bunch of Squid + Tor.
I already wrote about how to make Squid transparent with HTTPS proxying without certificate substitution. And of course, I tried to implement the idea on it. But I was disappointed. HTTP requests were fine with TOR, but no HTTPS. The problem is not very well known, and I learned from one of the developers that this is a flaw in older versions of Squid. But during the experiments, a solution was found - Squid 3.5.27, which fixed this bug + beautiful domain names in the logs (https), instead of ip addresses. But even here I was waited by several disappointments, which will be discussed below. But everything, as they say, is finished with a file.
So, the source data:
- Debian Stretch (9) x86 (I haven't tried x64)
- Sorts Squid 3.5.23 from repository
- Squid from a fresh site
- Openssl
- Libecap3
- Tor
- Privoxy
- Straight arms and lots of coffee with cookies.
Build Squid with pens or install ready-made packages (links below), you decide. If you think that I’ve seen
So prepare for the build:
apt-get install fakeroot build-essential devscripts apt-get build-dep squid3 apt-get install libecap3 apt-get install libecap3-dev apt-get install libssl1.0-dev apt-get install libgnutls28-dev IMPORTANT!
It is very important to install libssl1.0-dev, and not another version, otherwise Squid will either lag, or fail to build at all due to incomprehensible errors.
Next we get the source code for Squid 3.5.23
apt-get source squid3 We download this particular archive with Squid sources:
wget -O squid-3.5.27-2018.tar.gz http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.27-20180318-r1330042.tar.gz Go to the Squid source directory and update the source code to the newly added sorts:
cd squid3-3.5.23/ uupdate -v 3.5.27-2018 ../squid-3.5.27-2018.tar.gz Go to the new directory with updated sources:
cd ../squid3-3.5.27-2018 Add to debian / rules options for compiling:
--enable-ssl \ --enable-ssl-crtd \ --with-openssl The board
You can, by the way, cut down unnecessary options for you, this will speed up the compilation
Next, you need to patch the source code with this patch:
<b>client_side_request.patch</b> --- src/client_side_request.cc Thu Aug 18 00:36:42 2016 +++ src/client_side_request.cc Mon Sep 19 04:41:45 2016 @@ -519,20 +519,10 @@ // note the DNS details for the transaction stats. http->request->recordLookup(dns); - if (ia != NULL && ia->count > 0) { - // Is the NAT destination IP in DNS? - for (int i = 0; i < ia->count; ++i) { - if (clientConn->local.matchIPAddr(ia->in_addrs[i]) == 0) { - debugs(85, 3, HERE << "validate IP " << clientConn->local << " possible from Host:"); - http->request->flags.hostVerified = true; - http->doCallouts(); - return; - } - debugs(85, 3, HERE << "validate IP " << clientConn->local << " non-match from Host: IP " << ia->in_addrs[i]); - } - } - debugs(85, 3, HERE << "FAIL: validate IP " << clientConn->local << " possible from Host:"); - hostHeaderVerifyFailed("local IP", "any domain IP"); + debugs(85, 3, HERE << "validate IP " << clientConn->local << " possible from Host:"); + http->request->flags.hostVerified = true; + http->doCallouts(); + return; } void What is it for? I will explain. When I wrote the first article about peek and splice, I said that newer versions do not work, and it was like that, and this patch corrects the very problem that Squid selectively tears off HTTPS connections with an interesting By posting to cache.log:
SECURITY ALERT: Host header forgery detected on ... (local IP does not match any domain IP) The fact is that on one host something resolves in one IP, on a neighboring one sometimes in another, on Squid itself in the third, since There is a DNS cache and it is not updated synchronously. Squid does not match the ip-domain in its cache (because it updated its cache a little earlier or later) and terminates the connection. It seems like protection, but nowadays this is considered normal (round-robin DNS). Developers reinsured. And we do not need it at all! To those who say that this patch probably carries a security risk, I will answer that I have consulted on this patch with Yuri Voinov, who is directly related to the Squid development team. There is no threat here!
So, the file for the patch was created, the code was dropped, you need to patch:
patch -p0 -i client_side_request.patch Next, you need to cancel the application of one patch when compiling (otherwise you will receive an error that this patch cannot be applied, since it has already been applied). Go to debian / patches / series and comment out there 0003-SQUID-2018_1.patch , putting a # sign in front of it:
#0003-SQUID-2018_1.patch Well, then - compile and build packages!
dpkg-buildpackage -us -uc -nc Install squid-langpack
apt-get install squid-langpack and install fresh packages
dpkg -i squid-common_3.5.27-2018-1_all.deb dpkg -i squid_3.5.27-2018-1_i386.deb dpkg -i squid3_3.5.27-2018-1_all.deb If apt curses dependency, do
apt-get -f install Next, you need to turn off Squid from autoload (by default, the init file is used, Squid complains that the PID file is unavailable)
systemctl disable squid and create a systemd service in the / etc / systemd / system directory (the service file is in the source code, and completely copied here)
cat /etc/systemd/system/squid3.service ## Copyright (C) 1996-2018 The Squid Software Foundation and contributors ## ## Squid software is distributed under GPLv2+ license and includes ## contributions from numerous individuals and organizations. ## Please see the COPYING and CONTRIBUTORS files for details. ## [Unit] Description=Squid Web Proxy Server After=network.target [Service] Type=simple ExecStart=/usr/sbin/squid -sYC -N ExecReload=/bin/kill -HUP $MAINPID KillMode=process [Install] WantedBy=multi-user.target Turn it on
systemctl enable squid3.service Install tor, privoxy
apt-get install tor privoxy I personally did not touch the Tor configuration at all, but the Privoxy config can be brought to this form:
listen-address 127.0.0.1:8118 toggle 0 enable-remote-toggle 0 enable-remote-http-toggle 0 enable-edit-actions 0 forward-socks5t / 127.0.0.1:9050 . max-client-connections 500 Almost done. Go to the directory / etc / squid , something we change there. Create a pem file for splice:
openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout squidCA.pem -out squidCA.pem And we will bring squid.conf to the following form:
acl localnet src 192.168.0.0/24 # acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT acl SSL method CONNECT # DNS Squid. DNS dns_nameservers 77.88.8.8 # , Tor acl rkn url_regex "/etc/squid/tor_url" http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localnet http_access allow localhost http_access deny all icp_access deny all htcp_access deny all # intercept http_port 192.168.0.1:3128 intercept options=NO_SSLv3:NO_SSLv2 # , # , , , # , , # , =) http_port 192.168.0.1:3130 options=NO_SSLv3:NO_SSLv2 # , HTTPS https_port 192.168.0.1:3129 intercept ssl-bump options=ALL:NO_SSLv3:NO_SSLv2 connection-auth=off cert=/etc/squid/squidCA.pem sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER # ( .domain.com) acl blocked ssl::server_name "/etc/squid/blocked_https.txt" acl step1 at_step SslBump1 ssl_bump peek step1 # , ssl_bump terminate blocked ssl_bump splice all # , never_direct allow rkn # , , - Privoxy cache_peer 127.0.0.1 parent 8118 0 no-query no-digest default cache_peer_access 127.0.0.1 allow rkn cache_peer_access 127.0.0.1 deny all sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 logfile_rotate 4 pid_filename /var/run/squid.pid The url_regex list looks like this (the list is given for an example!):
zenway\.ru \.*google\.com \.*viber\.* \.amazon\.com \.fbcdn\.net \.slack\.* media\.api\.viber\.com* static\.viber\.com* secure\.viber.* \*.cloudfront\.net fonts\.gstatic\.com med-edu\.ru Comment
You can use and not regex, you can use dstdomain, you can also "screw" lists with improperly blocked subnets (dst)
For more information about this format of the list, read in the documentation. The result did not make me wait long - it all worked as planned. And it works to this day. Perhaps the article was late, but it will probably come in handy in the future).
Topic will replenish whenever possible.
Ready Squid Packages (.deb x86)
UPD 04.05.18:
HTTPTunnelPort 8118
Found a bug. Using HTTPTunnelPort is not recommended, Privoxy is needed until — until this is closed. Thanks to Yuri Voinov!
Also corrected the article on the part of libecap3. You do not need to compile anything, but simply install it from the Stretch repository. Thanks AlucoST for the comment
UPD 05/05/18: Comrade dartraiden suggests that you can add to the Tor config
ExcludeExitNodes {ru}, {ua}, {by} This excludes the use of output nodes in specified countries.
Many thanks to Yuri Voinov , who helped in solving problems with the work of this bunch!
Source: https://habr.com/ru/post/354708/
All Articles