Filtering the blacklist of sites by URL

This article describes a bunch of Squid + TPROXY standing on a separate machine. As it turned out, this topic is almost not covered on the Internet.

By occupation, the task of filtering the black list of sites from ROSCOMNADZOR arose. During the audit, we were threatened that if we did not do it, they would impose sanctions. It is said to have done (just a few minutes) a stupid and not very friendly to our customers implementation, which simply did not give access to all blacklisted sites. At the same time filtering was just by IP. And naturally blocked access to all sites on the IP.

It was necessary to redo it, and even competently redo it. Firstly, this is not beautiful / sloppy, and secondly, the task itself is very interesting.

In this article I will not tell you how to set up an automatic retrieval of this list. I can only say one thing, I did not bother with the constant signing of the request, and did it once on a computer with installed software and a key.
')

Let's start with the machine with Squid3 installed.

Almost all settings were taken from the official Squid + TPROXY

ip -f inet rule add fwmark 1 lookup 100 ip -f inet route add local default dev eth0 table 100 

 cat /etc/sysctl.conf: .... net.ipv4.ip_forward = 1 net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.eth0.rp_filter = 0 

So that we can determine which traffic has already passed the test, but why not we create a separate VLAN, in our case 5, for outgoing packets. Let me remind you that in the TPROXY IP mode, the source address does not change.

 default via 192.168.70.2 dev eth0.5 192.168.1.35/25 dev eth0 proto kernel scope link src 192.168.1.36 192.168.70.0/30 dev eth0.5 proto kernel scope link src 192.168.70.1 

Iptables rules

 iptables -t mangle -N DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT 

To prevent existing connections from falling into the rule TPROXY

 iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT 

Actually the rule TPROXY

 iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129 

Basic settings.

 http_port 3129 tproxy disable-pmtu-discovery=off ... acl bad_urls url_regex "/etc/squid3/bad_hosts.list" .... http_access deny bad_urls http_access allow localnet deny_info http://www.somehost.ru bad_urls ... 

The last line in the config will make REDIRECT to the specified site if the user requests one of the prohibited sites.

The rest of the settings in squid3 to taste.

Settings on the router

These settings we need to apply to the router.

 cat /etc/sysctl.conf: .... net.ipv4.ip_forward = 1 net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.eth0.rp_filter = 0 

Naturally enabled NAT gray subnets.

Create a chain in which we will filter

 iptables -t mangle -F DIVERT 

In this chain, we will mark all packages for which the outgoing or destination port is 80 (only when writing the article I realized that the ports were superfluous, but I left them, all of a sudden it is easier for someone to understand)

 iptables -t mangle -A DIVERT ! -i eth1.5 -p tcp --dport 80 -j MARK --set-mark 1 iptables -t mangle -A DIVERT ! -i eth1.5 -p tcp --sport 80 -j MARK --set-mark 1 

We’ll send all packets to this chain for which the source addresses or the destination 80 are in this case, we did not receive these packets from 5 VLANs (can someone tell me why in this case the connection is not traced and you have to do a reverse check)

 iptables -t mangle -A PREROUTING ! -i eth1.5 -p tcp -m set --match-set badip dst -m tcp --dport 80 -j DIVERT iptables -t mangle -A PREROUTING ! -i eth1.5 -p tcp -m set --match-set badip src -m tcp --sport 80 -j DIVERT 

Create a rule that in case there is a label on the packet that sends it to 101 in the routing table

 ip rule add fwmark 1 lookup 101 

Actually the routing table itself

 ip route list table 101 default via 192.168.1.35 dev eth1.3 

And lastly. That's how we have a list of bad sites.

 create badip hash:ip ipset flush badip ipset add badip 111.111.111.111 ipset add badip 2.2.2.2 ... 

Lists of blocked sites on both machines are loaded automatically, but they are interpreted differently.

Packet Motion Scheme

Blue - outgoing packet line
Green - reverse package.