Software routing: history of moving from hub-n-spoke (vyatta + openvpn) to fullmesh (mikrotik + tincvpn) - part 1

Under the cut of a curious hacker, a description of the sufferings of people who once saved money on cisca with dmvpn is waiting for them.
And also what came out of it.

Prehistory question

A few years ago, when the grass was greener, and the% company_name% was just starting up and had 2 branches,
it was decided to use Vyatta Network OS as a single routing platform, and static openvpn site-to-site tunnels as a solution for VPN.

One reason for this decision was the rich possibility of customizing ISRs.
As one of the backup measures, 2 routers were installed in each filal.
With this amount of peers, the "star" topology was considered justified.
')
As a hardware appliance was supposed to use
- full servers with esxi - for large installations
- servers with atom - for small ones.

SUDDENLY, for the year the number of branches increased to 10, and in another one and a half - to 28.
The constant stream of corporate bouts from businesses provoked the growth of traffic and the emergence of new network and application services.
Some of them settled on Linux virtual routers, instead of individual servers.

Vyatta functionality was not enough, a large number of packages affected the overall stability and performance of services.

Also, there were a number of problems with VPN.
1. At that time there were about 100 static tunnels terminated at one point.
2. In the future, it was still opening 10 branches.

In addition, openvpn in the current implementation,
1. does not allow creating dynamically generated site-to-site tunnels.
2. It is not multi-threaded, which on router processors with 1-2 uplink tunnels leads to overload of 1-2 cores (out of 8-16 in our case) with encryption and leads to hilarious consequences for traffic inside the tunnel.
3. Clause 2 is particularly characteristic of routers whose processors lacked support for AES-NI.

It was possible to use gre + ipsec, which would eliminate performance problems - but there would remain tunnels, hundreds of them.

It was very sad, I wanted to overlap with warm tsiska with hardware encryption and use dmvpn.
Or already global MPLS VPN.
In order not to trifle.

We had to do something (c)

- 1-2x processor servers with a tasty amount of brain and esxi were on board at the branches
- routers were decompressed and all third-party services moved to dedicated virtual machines with pure linux
- after some searches, the virtualized x86 routeros was chosen as the routing platform.
The operating experience of the micro-device had already been by that time, there were no problems with performance and stability on our tasks under x86.
On iron solutions, especially - on RB7 ** and RB2011 ** - it was and is more fun

Potentially, this solution gave an increase in the functional in
- route redistribution
- pbr + vrf
- mpls
- pim
- qos
as well as some others.

The problem was that routeros (like vyatta) does not support multipoint vpn.
It became sad again, they began to dig towards pure Linux fullmesh solutions.

Separately, by the way, thanks to Habrawser ValdikSS for his article .

Considered: peervpn, tinc, campagnol, GVPE, Neorouter, OpenNHRP, Cloudvpn, N2N
As a result, we stopped at tincvpn, as a compromise solution.
I also really liked gvpe using exotic encapsulation methods, but it didn’t allow specifying several IPs (from different ISPs) for remote peer.

What a shame, tinc was also single-threaded.
The solution was a dirty hack:
1. Between every two members, the mesh is raised not by 1, but by N tunnels, which are then combined into a bond interface with outgoing traffic balancing.

2. as a result, we obtain already N processes, which can already be distributed among different cores.

When using the specified scheme, in lab with
- 8 tinc-interfaces, aggregated into one bond according to the specified scheme
- forced binding of processes to different cores
- 1 x xeon 54 **, 55 ** (old generation processors without aes-ni support)
- aes128 / sha1,
- udp as a transport protocol.
- without using qos both inside and outside the tunnel.

The performance of vpn was about 550-600 mbps on a gigabit channel.
We were saved.

What happened:
1. The final solution received the working name "vpn-pair."
2. It consists of two virtual machines located on one esxi host and connected by a virtual interface.
1 - routeros, which deals exclusively with routing issues
2 - pure linux with a licked and slightly doped tinc.
Performs the role of vpn-bridge.
The tinc configuration is similar to the laboratory configuration, while the bond interface is sbrigen with a virtual link to the microtic.
As a result, all routers have a common L2, you can raise ospf and enjoy.
3. Each branch has 2 vpn-pairs.

... to be continued ...

Notes

1. In principle, it is possible to integrate tinc into vyatta, a patch was even written about this (for the version with 1 interface)
But to predict the behavior of such a bundle in an emergency (or after updating the platform) with a custom patch is completely difficult and for such a large network such experiments are undesirable.

2. Considered also the option "matryoshka".
He had 2 directions:
- purchase of hardware x86 microtics and support for linux-vm with a vpn-bridge using built-in virtualization (KVM).
did not take off due to poor performance.

- nested virtualization (esxi -> routeros -> kvm -> linux).
did not take off for the same reasons, due to the lack of support for VT-X / EPT emulation in esxi 5.1 (needed to start the kvm-machine).