Combined load balancing online channels

Prehistory

Sooner or later, the system administrator is faced with the need to distribute traffic across multiple channels, while naturally the desire is that each channel be used to the maximum. Faced with a similar need, and deciding not to reinvent the wheel, turned to the help of search engines. Since I have a server on Ubuntu, I turned my attention to the article http://help.ubuntu.ru/wiki/ip_balancing . I implemented “Method 1”, but during the test the following critical problems were noticed: when using links on some sites they did not open (for example, when trying to turn on music on the VKontakte resource). The reason is obvious - the request went through another channel. Having considered the situation, I decided to combine the approach to balancing. The logic is simple - most torrents and similar programs eat up traffic, so we share traffic. As a result, we distribute traffic with ports up to 11000 approximately evenly by the number of subscribers - subnets; with traffic with ports 11000-60000, we equalize the load on the channels.

Settings

It is assumed that the routing tables for each of the channels are created, let's call them chan1 , chan2 , chan3 - three channels, respectively.
Somewhere, for example, in /etc/rc.local we add something like:
')

ip rule add prio 101 fwmark 1 table chan1 ip rule add prio 102 fwmark 2 table chan2 ip rule add prio 103 fwmark 4 table chan3 

Create the /etc/rc.balance script:

 #!/bin/bash lst='/etc/rc.balance.lst' ########### Flushing ################## /sbin/iptables -t mangle -F PREROUTING /sbin/iptables -t mangle -F POSTROUTING /sbin/iptables -t mangle -F OUTPUT ####################################### /etc/rc.baltor /sbin/iptables -t mangle -A PREROUTING -d 172.16.0.0/16 -j RETURN /sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -m state --state INVALID -j DROP while read net mark do /sbin/iptables -t mangle -A PREROUTING -s $net -m state --state new,related -j CONNMARK --set-mark $mark done<$lst /sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -p udp --sport 11000:60000 --dport 11000:60000 -m state --state new,related -j BALANCE /sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -p tcp --sport 11000:60000 --dport 11000:60000 -m state --state new,related -j BALANCE /sbin/iptables -t mangle -A PREROUTING -s 172.16.0.0/16 -j CONNMARK --restore-mark exit 0 

Create a list of distribution of grids by channels (in the second column - the brand):

 172.16.0.0/22 1 172.16.4.0/22 2 172.16.8.0/22 4 

Script / etc / rc.baltor - balancing rules:

 #!/bin/bash /sbin/iptables -t mangle -F BALANCE lst='/etc/rc.cnload.lst' mrk=1 while read kld do /sbin/iptables -t mangle -A BALANCE -j CONNMARK --set-mark $mrk /sbin/iptables -t mangle -A BALANCE -m statistic --mode random --probability 0.$kld -j RETURN mrk=`expr $mrk \* 2` done < $lst /sbin/iptables -t mangle -A BALANCE -j CONNMARK --set-mark $mrk exit 0 

The /etc/rc.cnload script is a calculation of the probability depending on the channel load:

 #!/bin/bash cn1=800000 #        cn2=600000 #  .. cn3=400000 #   if1='eth1' #    if2='eth2' #  if3='eth3' #  lst='/etc/rc.cnload.lst' a1=`ifconfig $if1 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'` a2=`ifconfig $if2 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'` a3=`ifconfig $if3 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'` sleep 20 b1=`ifconfig $if1 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'` b2=`ifconfig $if2 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'` b3=`ifconfig $if3 | grep "RX bytes" | awk '{print $2}' | awk -F: '{print $2}'` c1=`expr \( $b1 - $a1 \) \* 8 / 20000` c2=`expr \( $b2 - $a2 \) \* 8 / 20000` c3=`expr \( $b3 - $a3 \) \* 8 / 20000` d1=`expr \( $cn1 - $c1 \) \* 100 / $cn1` d2=`expr \( $cn2 - $c2 \) \* 100 / $cn2` d3=`expr \( $cn3 - $c3 \) \* 100 / $cn3` #         60% if [ $d1 -lt "40" -o $d2 -lt "40" -o $d3 -lt "40" ] then e1=`expr 100 \* $d1 / \( $d1 + $d2 + $d3 \)` e2=`expr 100 \* $d2 / \( $d2 + $d3 \)` f1=`head -n 1 $lst | tail -n 1` f2=`head -n 2 $lst | tail -n 1` #     if [ $e1 -ne $f1 -a $e2 -ne $f2 ] then echo $e1 > $lst echo $e2 >> $lst /etc/rc.baltor fi fi exit 0 

Add to /etc/rc.local

 /etc/rc.balance 

and in / etc / crontab

 */1 * * * * root /etc/rc.cnload 

It is important that at the time of launch there already existed a file with coefficients /etc/rc.cnload.lst, it can be composed by running the script /etc/rc.cnload.

Conclusion

This method is successfully implemented in a network with 8000 subscribers. In addition to balancing, dynamic shaping is used, but this is a topic for another article.

All balance in everything.