"Gentle" balancing between multiple providers on the office gateway

This article describes the configuration of a Linux gateway for balancing traffic between channels of different providers.


The result achieved in this manual differs from the result of such manuals : the same external IP address is used for each client, which eliminates problems with Internet services that are not ready to change the client's IP address in one session.

Problems

For an ordinary consumer who does not have any of his own block of addresses and does not participate in the exchange of routes at the level of telecom operators, access to the Internet is possible only from those addresses that each Internet provider provides for each channel. This means that any connection of several channels from “domestic” Internet providers to a single node requires not only a choice between different upstream gateways, but also a choice of the correct outgoing address for communicating with them.

This state of affairs runs counter to how the standard routing mechanism works. In general, the route in the routing table depends only on the destination address and does not change the sender's address. Therefore, the first task that needs to be solved is to organize the correspondence of the sending interface, the gateway and the outgoing address when forwarding packets from clients.
')
The second problem to be solved is how to intelligently distribute the load from clients.

Balancing using multipath routes (as in the link at the beginning of the article) is carried out at the destination address and does not always look like an attractive solution. Where it would be worthwhile to have the same external address, the client may have a different session in one session and vice versa - when accessing popular resources, all clients will use one common channel.

Balancing at the sender address can be a compromise between compatibility and granularity of distribution. Each computer in the local network will always have the same external address and with a considerable number of computers in the network we will get a tolerable load distribution across the channels. Such a division can be achieved if we form hashes from local addresses, divide the entire space of hashes in proportion to the weights of the channels and use the value of this hash to select the channel.

Instruments

In the Linux kernel, it is possible to use several different tables at the same time, depending on what criteria the package meets. Often this mechanism is called policy based routing (PBR). This mechanism is controlled through the iproute2 toolkit.

Somewhat simplifying the formulation of the problem, one can say: all that is required of us to distinguish between routes, interfaces and outgoing addresses is to create for each provider an additional routing table that looks as if this channel of this provider were the only one. Then you need to add rules (the same routing policies) that trigger the appropriate routing table for the corresponding traffic.

As for balancing by sender, for this you can use the kernel network filter - netfilter (iptables). With the HMARK action, we will mark the packet with a hash of the address. According to the fwmark criterion in the routing rules (ip rule command), we will forward the packet to the desired routing table. iproute2 and iptables play well in pairs here.

Example

As an example, I’ll take a Debian based gateway that provides Internet access through three channels from two different providers. Such an example was chosen specifically, as it considers a delicate situation when the same upstream gateway of one provider is accessible through 2 different wires. The described configuration will have a similar appearance in all other debian-like operating systems, including Ubuntu.

The gateway has 4 working interfaces:

Int.	Description	Capacity	Address	Gateway
eth0	the local network		10.0.0.1/16	-
eth1	first channel of the first provider	100 Mbps	100.1.1.92/24	100.1.1.1
eth2	the only channel from the second provider	80 Mbps	200.2.2.22/24	200.2.2.1
eth3	second channel of the first provider	100 Mbps	100.1.1.93/24	100.1.1.1

Each channel is connected via a normal ethernet with a static address.

Initial state

Suppose we have already configured access through one channel. The / etc / network / interfaces file is:

As you can see, only one of the interfaces has a default route.

In /etc/sysctl.conf, the value of net.ipv4.ip_forward is set to 1. The iptables-persistent package is installed and the contents of the /etc/iptables/rules.v4 file are:

Only three rules for address translation from packets originating through three external interfaces. It does not matter to use a masquerade, but I stopped at it. Since the gateway is currently configured on only one interface, only the first rule actually works.

Routing rules

Now the most difficult: we will solve the first problem of matching addresses, interfaces and gateways. We will prepare three additional routing tables that we will use for different providers.
Let's start with the optional step - in order not to mention the routing tables everywhere by their number, let's get them in the file of correspondences of the numbers and names of the tables / etc / iproute2 / rt_tables:

So far, quite simple. Now fill these tables with routes. Let each table for each external channel contain all routes of all interfaces, except routes through other external interfaces. When picking up each external interface, add a rule that instructs to use the specified additional routing table if the sender address is equal to the address on this interface.

Note: the main routing table still gets only one gateway - all the gateway directives, except for one, are commented out.
Here, additional routes and rules are decorated with up directives that simply execute the command when raising the interface. Commands for adding routes and rules are grouped under the interfaces through which they can be implemented - this is most clearly seen in the example of the lo interface.

After restarting the network, we can notice how the output of the list of rules in the ip rule list command has changed:

The corresponding table can be viewed with the command “ip ro li table XXX”.

Already in this position, the server is ready to use all interfaces at once. We will verify this by trying to use all the interfaces one by one and checking the address observed by the outside:

If the external interfaces have real “white” addresses, then you can try to join the server using any address — the server will communicate through each of them through the correct channel and the correct provider. If so, then the most difficult thing behind.

Balancing

In my example there are two channels of 100 Mb / s and one 80 Mb / s. In order to divide the load equally between them, it is enough for me to divide it into 14 parts and send 5 parts to two 100-megabit channels and 4 parts to an 80-megabit channel. Add a rule to the firewall, marking the packet with numbers from 10,000 to 10,013, depending on the outgoing IP address:
# iptables -t mangle -A PREROUTING -s 10.0.0.0/16 -j HMARK --hmark-tuple src --hmark-offset 10000 --hmark-mod 14 --hmark-rnd 0xfeedcafe
# /etc/init.d/netfilter-persistent save

Simple enough. Parameters and values ​​speak for themselves. It now remains to send traffic, marked by different numbers, through different interfaces. As a result, / etc / network / interfaces takes the form:

For each interface, we added lines like “up ip rule add from 10.0.0.0/8 fwmark MARK table TABLE”. Each of them sends packets with a corresponding marking for routing to the specified table. In my example, numbers are intermixed between interfaces for the sake of uniformity.

Everything!

Optionally, scripts can be added to such a routing scheme, detecting the drop in one of the channels and rebalancing the load with this in mind. However, implementation details depend on taste and it is better to leave it to the reader.