Transparent bypass in home network
The latest news has once again highlighted the problem of blocking Internet resources. On the one hand , much has been written about ways to circumvent them , and once again it would seem to chew on this topic once again. On the other hand, to take some additional actions regularly to visit the necessary resource is not quite what the IT person should satisfy (and not always what a person can cope with to find a long time).
We need a simple and transparent solution for users, which, once set up, will allow us to simply use the Internet, without thinking about what has been blocked today at the request of the next copyist plagiarists .
By itself, it begs the idea of ​​bypassing the lock already on the home router.
Actually, it’s easy to pick up on a router and drive all traffic through VPN, and some VPN providers even have step-by-step instructions on how to configure OpenWrt to work with them.
')
But the speed of VPN services still lags behind the speeds of Internet access, and the VPN service either costs money, or has a lot of restrictions, or the need to regularly receive new logins. From the point of view of cost optimization, both financial and temporary, Tor looks preferable, but its speed is even worse, and it’s not the best idea to drive torrents through Tor.
Exit - redirect only blocked resources to VPN / Tor, passing the rest in the usual way.
Attention: this scheme does not provide anonymity of viewing blocked sites: any external link reveals your real IP.
A specific implementation on OpenWrt is provided at the end of the article. If you are not interested in the details and alternative solutions, then you can scroll right before her.
Setting up a VPN or Tor should not be difficult. Tor should be configured as a transparent proxy (or configure a bunch of tor and tun2socks ). Since the final goal is to bypass pkn locks, then in Tor's config it is advisable to prohibit the use of output nodes in the territory of the Russian Federation (
In Tor, the traffic is redirected by the rule with
For redirecting to VPN (or Tor + tun2socks), traffic is marked in the
In both cases, to classify traffic,
Formation of
Unfortunately, the option “drive all IPs from the registry” in ipset does not work as we would like: firstly, the lists do not contain all IP addresses of blocked hosts, secondly, in an attempt to escape from blocking, the resource’s IP address may change (and the provider he already knows this, but we don’t yet), well, in the third, there are false positives for sites located on the same shared hosting.
I do not really want to fuss with a dpi of one kind or another: after all, this should work on a rather weak hardware. The output is quite simple and to some extent elegant: dnsmasq (the DNS server that is most likely already installed on the router) can add ip-addresses to the corresponding ipset when resolving names (the option of the same name in the config). Just what you need: we put in the config all the domains that need to be unblocked, and further, if necessary, dnsmasq itself adds to ipset exactly the ip address that will be used to access the blocked resource.
I had doubts that dnsmasq would start and work normally with a config of half a dozen thousand lines (about as many entries in the registry after shrinking and tuning), but fortunately they turned out to be unfounded.
The fly in the ointment is that when updating the dnsmasq list you will have to restart, because by SIGHUP it config does not overload.
Should happen automatically as far as possible.
The first option (which is implemented in the example): create a list based on a single register of locks and update it by cron.
Roskomnadzor does not provide the general public with a register of locks, but the world is not without good people and there are at least two resources where it can be found. And, which is great, they also have an API. When parsing the list, you need to consider that in the list of domain names, in addition to the actual domain names, there are also IP addresses. They need to be processed separately (or even score on them: they are about 0.1% of the list and they are unlikely to lead to the resources you are interested in). Cyrillic domains are not always represented in punycode. A considerable part of the list is occupied by subdomains on the same second-level domain, there are domains with www / without www and simply duplicate entries. All of the above relates more to the list from rublacklist.net (it is also strange, in some places incorrectly, screened). It was for him that the monster lua-script (given below) had to be fenced in, normalizing and compressing the list almost twice. With antizapret.info, the situation is much better and one would be able to get along with an awk.
You can go the other way: many providers, when accessing a blocked resource, are redirected to the access restriction stub. For example
Now about another fly in the ointment: some providers have been noticed for the fact that in addition to the sites included in the list of sites, they are self-blocking and are not officially listed in the lists. At the same time, they block by sly and do not remove the plugs. So absolutely not to do without the manual drive.
ISP's DNS servers are naturally not worth being used as upstream servers. For blocking can occur at the stage of resource name resolution. Give the provider's server to the desired address, that this is CNAME block.mts.ru and that's it. The simplest solution is
When using Tor, you can get a bonus on surfing on .onion sites: Tor, when resolving a name via the built-in dns server, maps it to a virtual address from a predetermined range. Then you only need to redirect the call to this address to Tor's and voila's proxies. But once again I remind you that the connection with selective traffic tunneling does not provide anonymity.
The router itself should not be the worst, especially when using Tor. MIPS 400MHz @ 32MB RAM is the minimum worth considering.
If there is a USB port, the lack of a built-in flash can be compensated by a USB flash drive (in general, it seems to me a sensible idea not to use the built-in flash for regularly rewritable data).
Normally, OpenWrt firmware contains a stripped-down dnsmasq, which cannot ipset. You need to replace it with dnsmasq-full .
Of the packages that are not present by default, ipset , tor and tor-geoip are also required.
You also need either the luasocket package, or (in the strict saving flash mode) separately ltn12.lua in the / usr / lib / lua folder. To convert Cyrillic domains from utf8 to punycode, you need idn.lua to / usr / lib / lua and the luabitop package (or disable the options in the script config).
It remains to create the
We need a simple and transparent solution for users, which, once set up, will allow us to simply use the Internet, without thinking about what has been blocked today at the request of the next copyist plagiarists .
By itself, it begs the idea of ​​bypassing the lock already on the home router.
Actually, it’s easy to pick up on a router and drive all traffic through VPN, and some VPN providers even have step-by-step instructions on how to configure OpenWrt to work with them.
')
But the speed of VPN services still lags behind the speeds of Internet access, and the VPN service either costs money, or has a lot of restrictions, or the need to regularly receive new logins. From the point of view of cost optimization, both financial and temporary, Tor looks preferable, but its speed is even worse, and it’s not the best idea to drive torrents through Tor.
Exit - redirect only blocked resources to VPN / Tor, passing the rest in the usual way.
Attention: this scheme does not provide anonymity of viewing blocked sites: any external link reveals your real IP.
A specific implementation on OpenWrt is provided at the end of the article. If you are not interested in the details and alternative solutions, then you can scroll right before her.
Tunneling and forwarding traffic to the tunnel
Setting up a VPN or Tor should not be difficult. Tor should be configured as a transparent proxy (or configure a bunch of tor and tun2socks ). Since the final goal is to bypass pkn locks, then in Tor's config it is advisable to prohibit the use of output nodes in the territory of the Russian Federation (
<ExcludeExitNodes {RU} ).In Tor, the traffic is redirected by the rule with
REDIRECT to the transparent proxy port in the PREROUTING the nat netfilter table.For redirecting to VPN (or Tor + tun2socks), traffic is marked in the
mangle table, the label is then used to select the routing table that redirects traffic to the appropriate interface.In both cases, to classify traffic,
ipset used with hosts that are (once) blocked.Formation of ipset with (times) blocked hosts
Unfortunately, the option “drive all IPs from the registry” in ipset does not work as we would like: firstly, the lists do not contain all IP addresses of blocked hosts, secondly, in an attempt to escape from blocking, the resource’s IP address may change (and the provider he already knows this, but we don’t yet), well, in the third, there are false positives for sites located on the same shared hosting.
I do not really want to fuss with a dpi of one kind or another: after all, this should work on a rather weak hardware. The output is quite simple and to some extent elegant: dnsmasq (the DNS server that is most likely already installed on the router) can add ip-addresses to the corresponding ipset when resolving names (the option of the same name in the config). Just what you need: we put in the config all the domains that need to be unblocked, and further, if necessary, dnsmasq itself adds to ipset exactly the ip address that will be used to access the blocked resource.
I had doubts that dnsmasq would start and work normally with a config of half a dozen thousand lines (about as many entries in the registry after shrinking and tuning), but fortunately they turned out to be unfounded.
The fly in the ointment is that when updating the dnsmasq list you will have to restart, because by SIGHUP it config does not overload.
List domains
Should happen automatically as far as possible.
The first option (which is implemented in the example): create a list based on a single register of locks and update it by cron.
Roskomnadzor does not provide the general public with a register of locks, but the world is not without good people and there are at least two resources where it can be found. And, which is great, they also have an API. When parsing the list, you need to consider that in the list of domain names, in addition to the actual domain names, there are also IP addresses. They need to be processed separately (or even score on them: they are about 0.1% of the list and they are unlikely to lead to the resources you are interested in). Cyrillic domains are not always represented in punycode. A considerable part of the list is occupied by subdomains on the same second-level domain, there are domains with www / without www and simply duplicate entries. All of the above relates more to the list from rublacklist.net (it is also strange, in some places incorrectly, screened). It was for him that the monster lua-script (given below) had to be fenced in, normalizing and compressing the list almost twice. With antizapret.info, the situation is much better and one would be able to get along with an awk.
You can go the other way: many providers, when accessing a blocked resource, are redirected to the access restriction stub. For example
http://block.mts.ru/?host=/host/&url=/url/¶ms=/params/ . By address=/block.mts.ru/192.168.1.1 the same dnsmasq ( address=/block.mts.ru/192.168.1.1 ) with the A-record block.mts.ru to the address of the web server of the router (and placing a simple script on it), you can locally generate a list of the requested network users resources, add them to the dnsmasq config, re-do nslookup (so that the ip address is added to ipset) and redirect the user to the original URL again. But the need to restart dnsmasq every time is somewhat dampening. Yes, and will work only for http.Now about another fly in the ointment: some providers have been noticed for the fact that in addition to the sites included in the list of sites, they are self-blocking and are not officially listed in the lists. At the same time, they block by sly and do not remove the plugs. So absolutely not to do without the manual drive.
Additional Notes
ISP's DNS servers are naturally not worth being used as upstream servers. For blocking can occur at the stage of resource name resolution. Give the provider's server to the desired address, that this is CNAME block.mts.ru and that's it. The simplest solution is
server=8.8.8.8, server=8.8.4.4 . I haven’t yet seen modifications by providers of third-party DNS servers. If they start, you can send domain requests from the forbidden list to another upstream (via VPN / Tor), but without need I wouldn’t have inflated the config.When using Tor, you can get a bonus on surfing on .onion sites: Tor, when resolving a name via the built-in dns server, maps it to a virtual address from a predetermined range. Then you only need to redirect the call to this address to Tor's and voila's proxies. But once again I remind you that the connection with selective traffic tunneling does not provide anonymity.
Implementation on OpenWrt (15.05)
The router itself should not be the worst, especially when using Tor. MIPS 400MHz @ 32MB RAM is the minimum worth considering.
If there is a USB port, the lack of a built-in flash can be compensated by a USB flash drive (in general, it seems to me a sensible idea not to use the built-in flash for regularly rewritable data).
Normally, OpenWrt firmware contains a stripped-down dnsmasq, which cannot ipset. You need to replace it with dnsmasq-full .
Of the packages that are not present by default, ipset , tor and tor-geoip are also required.
You also need either the luasocket package, or (in the strict saving flash mode) separately ltn12.lua in the / usr / lib / lua folder. To convert Cyrillic domains from utf8 to punycode, you need idn.lua to / usr / lib / lua and the luabitop package (or disable the options in the script config).
Script update block lists
/usr/bin/rublupdate.lua
local config = { blSource = "antizapret", -- antizapret rublacklist groupBySld = 32, -- neverGroupMasks = { "^%a%a%a?.%a%a$" }, -- org.ru, net.ua neverGroupDomains = { ["livejournal.com"] = true, ["facebook.com"] = true , ["vk.com"] = true }, stripWww = true, convertIdn = true, torifyNsLookups = false, -- DNS TOR blMinimumEntries = 1000, -- , - dnsmasqConfigPath = "/etc/runblock/runblock.dnsmasq", ipsetConfigPath = "/etc/runblock/runblock.ipset", ipsetDns = "rublack-dns", ipsetIp = "rublack-ip", torDnsAddr = "127.0.0.1#9053" } local function prequire(package) local result, err = pcall(function() require(package) end) if not result then return nil, err end return require(package) -- return the package value end local idn = prequire("idn") if (not idn) and (config.convertIdn) then error("you need either put idn.lua (github.com/haste/lua-idn) in script dir or set 'convertIdn' to false") end local http = prequire("socket.http") if not http then local ltn12 = require("ltn12") end if not ltn12 then error("you need either install luasocket package (prefered) or put ltn12.lua in script dir") end local function hex2unicode(code) local n = tonumber(code, 16) if (n < 128) then return string.char(n) elseif (n < 2048) then return string.char(192 + ((n - (n % 64)) / 64), 128 + (n % 64)) else return string.char(224 + ((n - (n % 4096)) / 4096), 128 + (((n % 4096) - (n % 64)) / 64), 128 + (n % 64)) end end local function rublacklistExtractDomains() local currentRecord = "" local buffer = "" local bufferPos = 1 local streamEnded = false return function(chunk) local retVal = "" if chunk == nil then streamEnded = true else buffer = buffer .. chunk end while true do local escapeStart, escapeEnd, escapedChar = buffer:find("\\(.)", bufferPos) if escapedChar then currentRecord = currentRecord .. buffer:sub(bufferPos, escapeStart - 1) bufferPos = escapeEnd + 1 if escapedChar == "n" then retVal = currentRecord break elseif escapedChar == "u" then currentRecord = currentRecord .. "\\u" else currentRecord = currentRecord .. escapedChar end else currentRecord = currentRecord .. buffer:sub(bufferPos, #buffer) buffer = "" bufferPos = 1 if streamEnded then if currentRecord == "" then retVal = nil else retVal = currentRecord end end break end end if retVal and (retVal ~= "") then currentRecord = "" retVal = retVal:match("^[^;]*;([^;]+);[^;]*;[^;]*;[^;]*;[^;]*.*$") if retVal then retVal = retVal:gsub("\\u(%x%x%x%x)", hex2unicode) else retVal = "" end end return (retVal) end end local function antizapretExtractDomains() local currentRecord = "" local buffer = "" local bufferPos = 1 local streamEnded = false return function(chunk) local haveOutput = 0 local retVal = "" if chunk == nil then streamEnded = true else buffer = buffer .. chunk end local newlinePosition = buffer:find("\n", bufferPos) if newlinePosition then currentRecord = currentRecord .. buffer:sub(bufferPos, newlinePosition - 1) bufferPos = newlinePosition + 1 retVal = currentRecord else currentRecord = currentRecord .. buffer:sub(bufferPos, #buffer) buffer = "" bufferPos = 1 if streamEnded then if currentRecord == "" then retVal = nil else retVal = currentRecord end end end if retVal and (retVal ~= "") then currentRecord = "" end return (retVal) end end local function normalizeFqdn() return function(chunk) if chunk and (chunk ~= "") then if config["stripWww"] then chunk = chunk:gsub("^www%.", "") end if idn and config["convertIdn"] then chunk = idn.encode(chunk) end if #chunk > 255 then chunk = "" end chunk = chunk:lower() end return (chunk) end end local function cunstructTables(bltables) bltables = bltables or { fqdn = {}, sdcount = {}, ips = {} } local f = function(blEntry, err) if blEntry and (blEntry ~= "") then if blEntry:match("^%d+%.%d+%.%d+%.%d+$") then -- ip - iptables if not bltables.ips[blEntry] then bltables.ips[blEntry] = true end else -- , FQDN . 2 ( bl TLD - :)) local subDomain, secondLevelDomain = blEntry:match("^([a-z0-9%-%.]-)([a-z0-9%-]+%.[a-z0-9%-]+)$") if secondLevelDomain then bltables.fqdn[blEntry] = secondLevelDomain if 1 > 0 then bltables.sdcount[secondLevelDomain] = (bltables.sdcount[secondLevelDomain] or 0) + 1 end end end end return 1 end return f, bltables end local function compactDomainList(fqdnList, subdomainsCount) local domainTable = {} local numEntries = 0 if config.groupBySld and (config.groupBySld > 0) then for sld in pairs(subdomainsCount) do if config.neverGroupDomains[sld] then subdomainsCount[sld] = 0 break end for _, pattern in ipairs(config.neverGroupMasks) do if sld:find(pattern) then subdomainsCount[sld] = 0 break end end end end for fqdn, sld in pairs(fqdnList) do if (not fqdnList[sld]) or (fqdn == sld) then local keyValue; if config.groupBySld and (config.groupBySld > 0) and (subdomainsCount[sld] > config.groupBySld) then keyValue = sld else keyValue = fqdn end if not domainTable[keyValue] then domainTable[keyValue] = true numEntries = numEntries + 1 end end end return domainTable, numEntries end local function generateDnsmasqConfig(configPath, domainList) local configFile = assert(io.open(configPath, "w"), "could not open dnsmasq config") for fqdn in pairs(domainList) do if config.torifyNsLookups then configFile:write(string.format("server=/%s/%s\n", fqdn, config.torDnsAddr)) end configFile:write(string.format("ipset=/%s/%s\n", fqdn, config.ipsetDns)) end configFile:close() end local function generateIpsetConfig(configPath, ipList) local configFile = assert(io.open(configPath, "w"), "could not open ipset config") configFile:write(string.format("flush %s-tmp\n", config.ipsetIp)) for ipaddr in pairs(ipList) do configFile:write(string.format("add %s %s\n", config.ipsetIp, ipaddr)) end configFile:write(string.format("swap %s %s-tmp\n", config.ipsetIp, config.ipsetIp)) configFile:close() end local retVal, retCode, url local output, bltables = cunstructTables() if config.blSource == "rublacklist" then output = ltn12.sink.chain(ltn12.filter.chain(rublacklistExtractDomains(), normalizeFqdn()), output) url = "http://reestr.rublacklist.net/api/current" elseif config.blSource == "antizapret" then output = ltn12.sink.chain(ltn12.filter.chain(antizapretExtractDomains(), normalizeFqdn()), output) url = "http://api.antizapret.info/group.php?data=domain" else error("blacklist source should be either 'rublacklist' or 'antizapret'") end if http then retVal, retCode = http.request { url = url, sink = output } else retVal, retCode = ltn12.pump.all(ltn12.source.file(io.popen("wget -qO- " .. url)), output) end if (retVal == 1) and ((retCode == 200) or (http == nil)) then local domainTable, recordsNum = compactDomainList(bltables.fqdn, bltables.sdcount) if recordsNum > config.blMinimumEntries then generateDnsmasqConfig(config.dnsmasqConfigPath, domainTable) generateIpsetConfig(config.ipsetConfigPath, bltables.ips) print(string.format("blacklists updated. %d entries.", recordsNum)) os.exit(0) end end os.exit(1) Dnsmasq settings
/etc/dnsmasq.conf
server=/onion/127.0.0.1#9053 ipset=/onion/onion conf-file=/etc/runblock/runblock.dnsmasq Add to dnsmasq / etc / config / dhcp section
list server '8.8.8.8' list server '8.8.4.4' list rebind_domain 'onion' Netfilter settings
Add to / etc / config / firewall
config ipset option name 'rublack-dns' option storage 'hash' option match 'dest_ip' option timeout '86400' config ipset option name 'rublack-ip' option storage 'hash' option match 'dest_ip' config ipset option name 'rublack-ip-tmp' option storage 'hash' option match 'dest_ip' config ipset option name 'onion' option storage 'hash' option match 'dest_ip' option timeout '86400' config redirect option name 'torify-blocked-dns' option src 'lan' option proto 'tcp' option ipset 'rublack-dns' option dest_port '9040' option dest 'lan' config redirect option name 'torify-blocked-ip' option src 'lan' option proto 'tcp' option ipset 'rublack-ip' option dest_port '9040' option dest 'lan' config redirect option name 'torify-onion' option src 'lan' option proto 'tcp' option ipset 'onion' option dest_port '9040' option dest 'lan' Add to /etc/firewall.user
cat /etc/runblock/runblock.ipset | ipset restore Tor Settings
/ etc / torrc
User tor PidFile /var/run/tor.pid DataDirectory /var/lib/tor ExcludeExitNodes {RU} VirtualAddrNetwork 10.254.0.0/16 # .onion AutomapHostsOnResolve 1 TransPort 9040 TransListenAddress 127.0.0.1 TransListenAddress 192.168.1.1 # LAN DNSPort 9053 DNSListenAddress 127.0.0.1 #AvoidDiskWrites 1 # OpenWrt /var RAM (tmpfs) , It remains to create the
/etc/runblock , manually launch the lua /usr/bin/rublupdate.lua script one-time, make sure that it worked without errors, add it to cron (a couple of times a day is enough) and forget about the Roskomnadzor. Well, until they start to block the torus, or sites that publish the registry).Source: https://habr.com/ru/post/270657/
All Articles