We connect branches in one network. Reducing Internet costs

Greetings habrazhitel, not so long ago I faced the task of connecting into a single network the branches of one small company scattered across Siberia. The main problem was that OpenVPN had to be forced to work on top of unstable PPPoE simultaneously letting all traffic through OpenVPN


The initial goal was to save money on Internet traffic in the branches because in remote areas, the price of unlimited ADSL with a width of 256kb / s cost about 7-10t.r. per month, and the internet was vital.
All the joy was that almost all branches had connections from one provider, in which the concept of local and peering traffic existed, and the main office had a dedicated broad Internet (another provider, but by chance he was loyal to the branch provider and he had peer-to-peer traffic "with a price of about 6 kopecks per megabyte).

')

1. proxy

The quickest solution was the usual cascade of proxy servers, and this was done because before all the branches distributed the Internet in their direct modem, then everyone had to select 1 system administrator, who would act as a gateway, the system managers were not gifts, who would give the 800th stump, someone 233, in general, who had that ... Although today it’s 4 7 tr. You can build a decent gateway, but the owner is a master, I want to speak at no cost!

On these gateways ubuntu 8.04 LTS was installed configured as a gateway, so that it plugged into the local network, into the modem and into the outlet, and everything worked right away. in many branches, admins could only click "Any key" on the user's keyboard, but it does not matter, it went, gradually 7 branches reconfigured their modems, and stuck gateways :)

Immediately raised the proxy cascade, sent http traffic there, but as we all know, HTTP traffic is just a certain% of the total traffic, switching to simpler tariffs, there was a saving, but conditional, because the negligent admin or user could for example pull through a torrent something weighty, which promised money for the affiliate ...

Along the way, other tasks appeared in the central office - transfer of a comparative postman, gateway, portal set up in 2002 and untouched since then, but this deserves a separate article ...
And while we are just interested in the network ...

2. OpenVPN

I saw this thing for the first time, there was a certain fear before the first acquaintance, then reading manuals and Internet, I rolled up my sleeves and put it on :)

2.1 Server
has 2 eth1 network adapters (192.168.5.x) - Local network and eth0 (real ip 111.111.111.111) Internet with a wide channel.

apt-get install openvpn

Next, create a server configuration file
touch /etc/openvpn/server.conf

when the system is booted, all VPN connections for which in the / etc / openvpn folder there are corresponding files with the .conf extension are automatically raised

I got it like this.

port 1194 #
proto udp #
dev tun #
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key # This file should be kept secret
dh /etc/openvpn/dh1024.pem
server 10.10.10.0 255.255.255.0 # vpn subnet
ifconfig-pool-persist ipp.txt # ip
push "route 192.168.5.0 255.255.255.0" # home
keepalive 10 120
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
log-append openvpn.log
verb 4
mute 20
client-to-client
client-config-dir /etc/openvpn/ccd #
route 192.168.0.0 255.255.255.0 # 1
route 192.168.1.0 255.255.255.0 # 2


Create a directory in which individual client settings will be stored:

mkdir /etc/openvpn/ccd

Now you need to create keys and certificates for encryption and authorization.

cd /usr/share/doc/openvpn/examples/easy-rsa/2.0
source ./vars
./clean-all
./build-ca

Now we will create a certificate and private key for the server:

./build-key-server server

Create a key for the client (if there are several clients, the procedure will have to be repeated):

./build-key client1

for each client must be specified a unique name (in this case client1).

if a new client is created after some time, the procedure will be as follows:

cd /usr/share/doc/openvpn/examples/easy-rsa/2.0
source ./vars
./build-key client2


Generating Diffie-Hellman parameters:

./build-dh

Place the following files in the / etc / openvpn / directory

* ca.crt
* server.crt
* dh1024.pem
* server.key

Create the file /etc/openvpn/ipp.txt

The configuration file of the client machine /etc/openvpn/client.conf was like this

remote 111.111.111.111 1194
client
dev tun
proto udp
resolv-retry infinite # this is necessary for DynDNS
nobind
user nobody
group nogroup
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client1.crt
key /etc/openvpn/client1.key
comp-lzo
verb 4
mute 20
redirect-gateway
#show-net-up
verb 4

Now you need to copy the generated client keys and the authoritative server certificate from the server to the / etc / openvpn / folder:
* ca.crt
* client1.crt
* client1.key

If there is a 192.168.1.x network behind the client, in order for the server to see it, you need to add a route to it to the server.

On the server, create the file / etc / openvpn / ccd / client1 with the following content:

iroute 192.168.1.0 255.255.255.0
# 2, 2
#push "route 192.168.100.0 255.255.255.0"
# OpenVPN
push "redirect-gateway def1"

This is where the most evil problem happened to me.

OpenVPN receiving directive
push "redirect-gateway def1"
(if there is a 'pull' in its configuration), the client does not delete the old route, but adds the following entries to the routing table:
0.0.0.0/1 via 192.168.231.5 dev tun0
128.0.0.0/1 via 192.168.231.5 dev tun0
and if openvpn goes via ethernet, then everything works and pleases the admin and users, but the great ppp likes to take this route.
0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 ppp0

And OpenVPN swears something like this
Jul 2 19:28:53 ino ovpn-client[14465]: NOTE: unable to redirect default gateway -- Cannot read current default gateway from system

The solution to this problem has been long and tedious, although it is on the surface. if this gateway ppp route indicates the gateway instead of 0.0.0.0 the real gateway, then OpenVPN sees this route and adds its own without any problems.

Therefore, I created a file
/etc/ppp/ip-up.d/routing
In which I wrote a small script. (I ask you not to kick with your feet, I am very lazy, and did not write a full-fledged script for defining a gateway and editing routes, but I made it as a simple crutch)
I would be very happy if someone offers a more logical, reliable, and universal method of editing routes on the fly.
So far, there is no certainty that everything will work 100% with ppp cliffs, but life will show, if that happens, I will correct the topic.

#! /bin/sh
# 222...
gw1=`ip route show | grep 222 | awk '{print $1}'`
# 0.0.0.0 0.0.0.0
route del default
#
route add -net default gw ${gw1} dev ppp0

Making it executable
chmod ug+x /etc/ppp/ip-up.d/routing

After that, the reboot, after some time the server stopped pinging on the external ipu, but began to respond on the inside - 10.10.10.26

Threat Please take into account that the firewall of the gateway and the server and the client must be corrected in order for the users to have a vital Internet.
For example, I did it like this:
-A POSTROUTING -s 192.168.0.0/255.255.0.0 -j MASQUERADE
There is no binding to either the external interface or ip :)
when, for various reasons, we don’t have openvpn, then users will have a direct Internet, and when it appears, all traffic will fly through it.

Conclusion

In life, this system is assembled in stages, first the server starts and the client starts, ping each other at 10.10.10.x, then routes are added to the networks that are behind the server and the client, they are checked when everything is stable and reliably we add the directive
push "redirect-gateway def1"
And again we are working and living, everything was done without leaving the office, on the gateways of the branches, network interfaces were signed, so that the administrator just plugged in the network cables and power, called me, and then I set up ssh on the working mode.

Oh yeah, I almost forgot, the most important thing is profit

In addition to the fact that now the whole network addresses the servers and services of the branches and the center by internal ip addresses, so also financial savings.

Previously, each branch spent an average of 7,000 rubles on the Internet. per month, now per month, each of them pays 550r. for access to the peering, the Internet is not consumed (except for the central one), for the beginning 7 branches were launched, then there will be more.
it turns out that for the year with the old scheme the company would spend 588 000 rub. on the Internet . , and with the current scheme, 46 200 rubles will be spent per year .

What's next?

And now, on this whole heh ... we will try to fly! I will try to deploy IP Telephony in order to minimize the cost of long-distance telephone calls between branches, to connect softATS with hardware in branches, which I will definitely write about. Good luck

update1 Many questions have arisen to the term “Large company” I will try to clarify.

It is large in Siberia, the center has a staff of 200 employees, branches from 20 to 60, 10-20 objects are still attached to each branch. 5-15 people. branches less than 30.

The company is not very agile, the main control comes from the state, the equipment, the company makes some steps towards IT development :)

Update2
And so, after testing, it turned out that if the control server was permanently lowered for a long time, then clients would not willingly raise openvpn and traffic could go straight to the expensive Internet.
Also, if the ADSL channel is forcibly broken, then it seems to be openvpn and it tries to start again, but it does not work for him, and he goes around in a circle.
I tried all sorts of options and keepalive and ping-restart and so on ... did not help ...

Therefore, we write a small script that will check the state of affairs.
touch /usr/bin/vpn_keepalive.sh

C content.

#! /bin/sh
# ,
#debug_out=/dev/null
debug_out=/dev/stdout
#NEXTHOP
# OpenVPN
NEXTHOP=192.168.5.1
#
OPEN_VPN_CMD="sudo /etc/init.d/openvpn restart"
PING=/bin/ping

logger_opts="-t $0"
if [ "$debug_out" = "/dev/stdout" ]
then
logger_opts="$logger_opts -s"
fi
pckts_rcvd=`$PING -c 8 -q -W 2 $NEXTHOP | grep transm | awk '{print $4}'`
echo "host: $NEXTHOP, pckts_rcvd: $pckts_rcvd" >$debug_out
if [ $pckts_rcvd -eq 0 ]
then
echo "Connection with $NEXTHOP lost, resetting" | logger $logopts
$OPEN_VPN_CMD > $debug_out
else
echo "Connection with $NEXTHOP up, no action" | logger $logopts
fi

Making it executable
chmod ug+x /usr/bin/vpn_keepalive.sh

The script pings the host, and if 0 packets are returned, it will execute the restart command.
after that, everything is guaranteed to rise, the correct routes are flowing in, and the traffic goes through VPN traffic

We throw this script in kroner
crontab -e

for example, every 2 minutes.
0-59/2 * * * * /usr/bin/vpn_keepalive.sh

And all if debug is enabled, then in the logs (syslog) it will be noted like this.
logger: Connection with 192.168.5.1 up, no action

Good luck!