Transparent Squid with HTTPS filtering resources without certificate substitution (x86)
It is no secret that in large offices the topic of filtering the Internet is quite relevant. A lot of software and hardware solutions cope with this task. But now all the sites that we cut earlier work over the HTTPS protocol, i.e. port 443. As is known, this protocol cannot be traced, listened to, etc., impossible. And any caching filtering proxy server, redirector, etc., filters only HTTP, i.e. port 80. How to cut Vkontakte, Odnoklassniki, iphide.info and many other similar sites? How to block access to personal mail in the organization, if the use of it is prohibited by the procedures in the organization? Yes, you can filter by IP addresses, but they often change, and on many resources there are several IP addresses. To block them at the firewall level is somehow not at all an Orthodox solution, and not at all convenient.
And so, quite recently, one comrade told me that he was raising a caching proxy with HTTPS filtering in his office, which interested me.
And he raised Squid 3.5.8 . As it turned out, it reworked the organization of interception of encrypted HTTPS sessions (ssl_bump), instead of interception modes (“modes”), actions (“actions”) were introduced, which can also be used in the ACL. The “server-first” mode, in which the connection to the target server is first established, and then a secure connection between the client and the proxy is created, is now available as an action “bump”. The “none” mode, in which a TCP tunnel is created without traffic decryption, is now available as an “splice” action.
For backward compatibility, the “peek-and-splice” action has been added, in which the decision on the type of connection created at the beginning (client-proxy or proxy server) is made on the basis of SSL hello messages. Added “peek” and “stare” actions to obtain a client or server certificate while retaining the possibility of further use of the “splice” and “bump” modes for connections. Added a "terminate" action to close connections to the client or server. That's just the time we need SSL BUMP, PEEK-and-SPLICE and Terminate. In general, the scheme of work is actually quite simple. Squid connects to the HTTPS resource, receives its certificate, and can “see” some data about the resource, in particular, the name of the server, which we just need to block! All manuals that are on the Internet, every now and then describe Man in the middle (MITM) attack with the substitution of certificates, in which, in principle, some websites and bank clients do not work, and users clearly see that they are being watched. Together with a friend, we jointly achieved a way of filtering and tracking HTTPS without replacing certificates, without MITM and others, and all this in a transparent mode without configuring browsers!
')
Subsequently, I encountered some difficulties, in particular, Squid started to segfolt under heavy load. But the problem was solved. The fact is that in Openssl there are some kind of bugs that are fixed in the library Libressl. Therefore, you must first integrate into the Libressl system, then after making the patch in the bio.cc file in the Squid source code, start compiling the latter. Go! I’ll make a reservation that I’m using the Debian Jessie x86 distribution, and I finally compiled Squid with version 3.5.9 (the latest version at the moment), and the article is intended for a more or less experienced Linux user, as some points are omitted, but only the most importantly, since I'm too lazy to chew everything.
To get started, let's get ready to build packages:
Do not forget to go to the folder where you will collect the source, so as not to zamra yourself Home. Next, download, compile and install Libressl:
We assemble and install, after which we reread library hashes:
Well, you need to configure the use of LibreSSL by default:
Check to see if Libressl was installed:
Happened!
After performing these steps, you need to edit sources.list, including the source code from the testing branch (this is necessary, since we need to compile a new libecap, which in turn is required to build Squid):
Update the package cache:
And now we download the necessary sources from Testing:
Next, collect libecap:
Remove the old one and install the new one:
Downloading thelatest and most recent Squid snapshot:
Update the Squid source code obtained earlier to the new ones, and then we will work in the directory with the new sources:
In order for all the buns we need to work, we need to compile Squid with the necessary options, so we’ll add the following compilation options to debian / rules:
Download the patch for bio.cc, which is needed for correct work with Libressl, from here bugs.squid-cache.org/attachment.cgi?id=3216 and apply it
Now you can start compiling and building Squid. But not so fast! Everything compiles without problems, at least on the x86 architecture, but at the very end, at the deb package build stage, you will be kindly told in the console: “ah ah ah, I can’t understand what dependencies are needed for libssl.so.32” (this version of the library from libressl). It’s understandable how Debian knows about it. We will cheat the system by specifying the option "do not check dependencies", like this:
And now let's get down to compiling and building:
After the build, install the language pack for Squid:
Next, install the newly created packages:
If the system zamatsya on unsatisfied dependencies, we will do:
Almost done. Go to the directory / etc / squid, something we change there. Create a pem file needed for SSL-Bump'ing:
And we will bring squid.conf to the following form:
Next, we wrap the necessary ports on Squid with the firewall:
All is ready! You can solemnly run Squid:
Let's see if everything is fine with Squid:
If there are no errors, you can work. Unfortunately, when blocking HTTPS resources, Squid's “Access Denied” message does not appear, but instead, the browser gives an error about the impossibility of creating a connection. If someone tells you how to do this, I will be very happy.
UPD: in the Squid version that I originally compiled, i.e. 3.5.9, an annoying bug (or feature) was found, due to which some HTTPS sites stop opening over time. Solution: compile version 3.5.8.
UPD2: created another bug report on the problem in 3.5.9, I will update the topic if something clears up.
UPD3: version 3.5.10 has been released with fixed bugs, at least the patch on the bio.cc file has already been applied there. Not tested yet
UPD4: edited the article a bit.
UPD5: direct link to download the archive with all the necessary packages (SQUID 3.5.8) , so far this is the only working version! The rest, which are versions above, have a bug that causes Squid to stop working with HTTPS transparent proxying. VERSION FOR X86 !!!
ATTENTION!!! To avoid misunderstandings, install version 3.5.8! There are no problems on it, and if everything is done exactly according to the article, then everything will work! If you put the version above, I do not guarantee the correct operation of transparent filtering!
Regular Update !!! Debian Testing src repository is now 3.5.10 !!! add squeeze src repository to sources.list. There is version 3.1, but that's okay, everything will be updated to version 3.5.8 as it should, only change the paths to directories in accordance with the version of the downloaded sources!
UPD 12/02/15: I apologize if I made someone crutch. I reloaded the squid 3.5.8 archive, there was one deb package missing! squid 3.5.8_all.
UPD 12/02/15: Attention! If you have a problem installing libressl, then you need to do this: first install openssl, and then install libressl (if you install the version from the archive with packages, do not forget to replace openssl with libressl, as in the article!). This will solve the problem of “not seeing” the library libssl.so.32
UPD 10.12.15: The next version of the article appeared, with a slightly different approach to compiling Kalmar and building packages. Right here
UPD 12/14/15: I want to share great news with my colleagues! I found a way to make new versions of Squid work without special dances with a tambourine! It is necessary that the clients and Squid settings have the same DNS! In my case, the Bind is spinning on the squid gateway. He appointed his clients, and Kalmar, with a directive:
UPD 12/14/15: I don’t know why, but at the moment on Debian 8.2 when building Squid using the method described above, I get a message stating that the mime.conf file was not found. And indeed, after all, in the Squid source code there is mime.conf.default. I found a solution. You need to edit two files:
1) debian / squid-common.install, and lead the string
in this view:
2) It is also necessary to make sure that after installing the package the file is renamed by the automaton into a “viewable view”, i.e. in mime.conf. To do this, edit the file debian / squid-common.postinst, bringing it to this form:
That is, add a line to rename the file, and leave the rest, as is, do not touch.
Everything, after this we form the packages, they will be formed and delivered.
UPD 06/19/17: since the linux-admin.tk domain was stolen from me, I had to create a new, already “normal” domain, so I changed the link to download the archive.
UPD 04.05.18: wrote a new article where the problem of glitches of new versions of Squid + is solved in the logs beautiful domain names instead of ip addresses + RKN bypass blocking.
And so, quite recently, one comrade told me that he was raising a caching proxy with HTTPS filtering in his office, which interested me.
And he raised Squid 3.5.8 . As it turned out, it reworked the organization of interception of encrypted HTTPS sessions (ssl_bump), instead of interception modes (“modes”), actions (“actions”) were introduced, which can also be used in the ACL. The “server-first” mode, in which the connection to the target server is first established, and then a secure connection between the client and the proxy is created, is now available as an action “bump”. The “none” mode, in which a TCP tunnel is created without traffic decryption, is now available as an “splice” action.
For backward compatibility, the “peek-and-splice” action has been added, in which the decision on the type of connection created at the beginning (client-proxy or proxy server) is made on the basis of SSL hello messages. Added “peek” and “stare” actions to obtain a client or server certificate while retaining the possibility of further use of the “splice” and “bump” modes for connections. Added a "terminate" action to close connections to the client or server. That's just the time we need SSL BUMP, PEEK-and-SPLICE and Terminate. In general, the scheme of work is actually quite simple. Squid connects to the HTTPS resource, receives its certificate, and can “see” some data about the resource, in particular, the name of the server, which we just need to block! All manuals that are on the Internet, every now and then describe Man in the middle (MITM) attack with the substitution of certificates, in which, in principle, some websites and bank clients do not work, and users clearly see that they are being watched. Together with a friend, we jointly achieved a way of filtering and tracking HTTPS without replacing certificates, without MITM and others, and all this in a transparent mode without configuring browsers!
')
Subsequently, I encountered some difficulties, in particular, Squid started to segfolt under heavy load. But the problem was solved. The fact is that in Openssl there are some kind of bugs that are fixed in the library Libressl. Therefore, you must first integrate into the Libressl system, then after making the patch in the bio.cc file in the Squid source code, start compiling the latter. Go! I’ll make a reservation that I’m using the Debian Jessie x86 distribution, and I finally compiled Squid with version 3.5.9 (the latest version at the moment), and the article is intended for a more or less experienced Linux user, as some points are omitted, but only the most importantly, since I'm too lazy to chew everything.
To get started, let's get ready to build packages:
apt-get install git fakeroot build-essential devscripts apt-cache policy squid3 apt-get build-dep squid3 apt-get build-dep libecap2 apt-get install libssl-dev libgnutls28-dev Do not forget to go to the folder where you will collect the source, so as not to zamra yourself Home. Next, download, compile and install Libressl:
wget http://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.1.6.tar.gz tar -xzvf libressl-2.1.6.tar.gz cd libressl-2.1.6 We assemble and install, after which we reread library hashes:
./configure make checkinstall --pkgname libressl --pkgversion 2.1.6 dpkg -i libressl_2.1.6-1_i386.deb ldconfig Well, you need to configure the use of LibreSSL by default:
mv /usr/bin/openssl /usr/bin/openssl-1 update-alternatives --install /usr/bin/openssl openssl /usr/bin/openssl-1 10 update-alternatives --install /usr/bin/openssl openssl /usr/local/bin/openssl 50 update-alternatives --config openssl Check to see if Libressl was installed:
openssl version LibreSSL 2.1.6 Happened!
After performing these steps, you need to edit sources.list, including the source code from the testing branch (this is necessary, since we need to compile a new libecap, which in turn is required to build Squid):
deb-src http://ftp.de.debian.org/debian/ testing main contrib non-free Update the package cache:
apt-get update And now we download the necessary sources from Testing:
apt-get source squid3/testing apt-get source libecap3/testing Next, collect libecap:
cd libecap-1.0.1/ dpkg-buildpackage -us -uc -nc -d Remove the old one and install the new one:
apt-get purge libecap2 dpkg -i libecap3_1.0.1-2_i386.deb dpkg -i libecap3-dev_1.0.1-2_i386.deb Downloading the
wget http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.8.tar.gz Update the Squid source code obtained earlier to the new ones, and then we will work in the directory with the new sources:
cd squid3-3.5.7/ uupdate -v 3.5.8 ../squid-3.5.8.tar.gz cd ../squid3-3.5.8/ In order for all the buns we need to work, we need to compile Squid with the necessary options, so we’ll add the following compilation options to debian / rules:
--enable-ssl --enable-ssl-crtd --with-openssl Download the patch for bio.cc, which is needed for correct work with Libressl, from here bugs.squid-cache.org/attachment.cgi?id=3216 and apply it
patch -p0 -i bug-4330-put_cipher_by_char-t1.patch Now you can start compiling and building Squid. But not so fast! Everything compiles without problems, at least on the x86 architecture, but at the very end, at the deb package build stage, you will be kindly told in the console: “ah ah ah, I can’t understand what dependencies are needed for libssl.so.32” (this version of the library from libressl). It’s understandable how Debian knows about it. We will cheat the system by specifying the option "do not check dependencies", like this:
export DEB_DH_SHLIBDEPS_ARGS_ALL=--dpkg-shlibdeps-params=--ignore-missing-info And now let's get down to compiling and building:
dpkg-buildpackage -us -uc -nc After the build, install the language pack for Squid:
apt-get install squid-langpack Next, install the newly created packages:
dpkg -i squid-common_3.5.8-1_all.deb dpkg -i squid_3.5.8-1_i386.deb dpkg -i squid3_3.5.8-1_all.deb dpkg -i squidclient_3.5.8-1_i386.deb If the system zamatsya on unsatisfied dependencies, we will do:
apt-get -f install Almost done. Go to the directory / etc / squid, something we change there. Create a pem file needed for SSL-Bump'ing:
openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout squidCA.pem -out squidCA.pem And we will bring squid.conf to the following form:
acl localnet src 192.168.1.0/24 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT dns_nameservers 8.8.8.8 http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localnet http_access allow localhost http_access deny all # intercept http_port 192.168.1.254:3128 intercept options=NO_SSLv3:NO_SSLv2 # , # , , , # , , # , =) http_port 192.168.1.254:3130 options=NO_SSLv3:NO_SSLv2 # , HTTPS https_port 192.168.1.254:3129 intercept ssl-bump options=ALL:NO_SSLv3:NO_SSLv2 connection-auth=off cert=/etc/squid/squidCA.pem always_direct allow all sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER # ( .domain.com) acl blocked ssl::server_name "/etc/squid/blocked_https.txt" acl step1 at_step SslBump1 ssl_bump peek step1 # , ssl_bump terminate blocked ssl_bump splice all sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache_dir aufs /var/spool/squid 20000 49 256 maximum_object_size 61440 KB minimum_object_size 3 KB cache_swap_low 90 cache_swap_high 95 maximum_object_size_in_memory 512 KB memory_replacement_policy lru logfile_rotate 4 I will tell a little about a config. The documentation on ssl_bump, peek-n-splice is quite extensive, but for our task you need to know the following. There are several stages of “handshaking” (i.e., handshake, interaction with the server). They are described in the documentation . We are interested in the example of Peek at SNI and Bump . That is, as the name implies, we look at the SNI information and bump the connection. Before this, we specify with the DONT_VERIFY_PEER option that we need to accept certificates even if they have not passed validation, and with the sslproxy_cert_error option we specify that we need to disable certificate checking on the server. The specified rule “acl step1 at_step SslBump1” is the first of the three possible ssl_bump steps. With this step, we get only SNI, and nothing more. This is enough for us. Then we use this ACL with the ssl_bump peek step1 string, that is, we are directly looking at the SNI, and after that we are blocking the connection if server_name from the list blocked is found in the SNI.
Next, we wrap the necessary ports on Squid with the firewall:
iptables -t nat -A PREROUTING -p tcp -m tcp -s 192.168.1.0/24 --dport 443 -j REDIRECT --to-ports 3129 iptables -t nat -A PREROUTING -p tcp -m tcp -s 192.168.1.0/24 --dport 80 -j REDIRECT --to-ports 3128 All is ready! You can solemnly run Squid:
systemctl start squid Let's see if everything is fine with Squid:
systemctl status squid ● squid.service - LSB: Squid HTTP Proxy version 3.x Loaded: loaded (/etc/init.d/squid) Active: active (running) since 2015-09-26 21:09:44 YEKT; 2h 6min ago Process: 1798 ExecStop=/etc/init.d/squid stop (code=exited, status=0/SUCCESS) Process: 1818 ExecStart=/etc/init.d/squid start (code=exited, status=0/SUCCESS) CGroup: /system.slice/squid.service ├─1850 /usr/sbin/squid -YC -f /etc/squid/squid.conf ├─1852 (squid-1) -YC -f /etc/squid/squid.conf ├─1853 (logfile-daemon) /var/log/squid/access.log └─1854 (pinger) 26 21:09:44 squid squid[1850]: Squid Parent: will start 1 kids 26 21:09:44 squid squid[1850]: Squid Parent: (squid-1) process 1852 started 26 21:09:44 squid squid[1818]: Starting Squid HTTP Proxy: squid. If there are no errors, you can work. Unfortunately, when blocking HTTPS resources, Squid's “Access Denied” message does not appear, but instead, the browser gives an error about the impossibility of creating a connection. If someone tells you how to do this, I will be very happy.
UPD: in the Squid version that I originally compiled, i.e. 3.5.9, an annoying bug (or feature) was found, due to which some HTTPS sites stop opening over time. Solution: compile version 3.5.8.
UPD2: created another bug report on the problem in 3.5.9, I will update the topic if something clears up.
UPD3: version 3.5.10 has been released with fixed bugs, at least the patch on the bio.cc file has already been applied there. Not tested yet
UPD4: edited the article a bit.
UPD5: direct link to download the archive with all the necessary packages (SQUID 3.5.8) , so far this is the only working version! The rest, which are versions above, have a bug that causes Squid to stop working with HTTPS transparent proxying. VERSION FOR X86 !!!
ATTENTION!!! To avoid misunderstandings, install version 3.5.8! There are no problems on it, and if everything is done exactly according to the article, then everything will work! If you put the version above, I do not guarantee the correct operation of transparent filtering!
Regular Update !!! Debian Testing src repository is now 3.5.10 !!! add squeeze src repository to sources.list. There is version 3.1, but that's okay, everything will be updated to version 3.5.8 as it should, only change the paths to directories in accordance with the version of the downloaded sources!
UPD 12/02/15: I apologize if I made someone crutch. I reloaded the squid 3.5.8 archive, there was one deb package missing! squid 3.5.8_all.
UPD 12/02/15: Attention! If you have a problem installing libressl, then you need to do this: first install openssl, and then install libressl (if you install the version from the archive with packages, do not forget to replace openssl with libressl, as in the article!). This will solve the problem of “not seeing” the library libssl.so.32
UPD 10.12.15: The next version of the article appeared, with a slightly different approach to compiling Kalmar and building packages. Right here
UPD 12/14/15: I want to share great news with my colleagues! I found a way to make new versions of Squid work without special dances with a tambourine! It is necessary that the clients and Squid settings have the same DNS! In my case, the Bind is spinning on the squid gateway. He appointed his clients, and Kalmar, with a directive:
dns_nameservers 127.0.0.1 . After that, everything worked successfully. Tested on Squid 4.0.3, compiled WITHOUT Libressl!UPD 12/14/15: I don’t know why, but at the moment on Debian 8.2 when building Squid using the method described above, I get a message stating that the mime.conf file was not found. And indeed, after all, in the Squid source code there is mime.conf.default. I found a solution. You need to edit two files:
1) debian / squid-common.install, and lead the string
etc/squid/mime.conf in this view:
etc/squid/mime.conf.default 2) It is also necessary to make sure that after installing the package the file is renamed by the automaton into a “viewable view”, i.e. in mime.conf. To do this, edit the file debian / squid-common.postinst, bringing it to this form:
#! /bin/sh set -e case "$1" in configure) mv /etc/squid/mime.conf.default /etc/squid/mime.conf ........... That is, add a line to rename the file, and leave the rest, as is, do not touch.
Everything, after this we form the packages, they will be formed and delivered.
UPD 06/19/17: since the linux-admin.tk domain was stolen from me, I had to create a new, already “normal” domain, so I changed the link to download the archive.
UPD 04.05.18: wrote a new article where the problem of glitches of new versions of Squid + is solved in the logs beautiful domain names instead of ip addresses + RKN bypass blocking.
I want to say thank you to comrade Dmitry Rakhmatullin, without him it would not have been possible to do what is written above. Also, a special thank you to the developers of Squid, who promptly responded to my bug report about the error in libssl. And thanks to the guys from Nadz Goldman and gmax007 from the Toaster, who sent my ideas on transferring Squid to a server that is physically separate from the main gateway and sent in the right direction.
Source: https://habr.com/ru/post/267851/
All Articles