📜 ⬆️ ⬇️

Load balancing with LVS

So, you have a loaded server and you suddenly wanted to unload it. You put and poured the same (same), but users persistently go to the first. In this case, of course, you need to think about load balancing.


The first thing that comes to mind is using the Round-robin DNS option. If anyone does not know, this is a method that allows you to blur requests between the n-th number of servers simply by giving a new IP to each DNS request.
What are the cons:

Although it is not worth adding between the input and the output “too much computer”, I want some methods of control over the situation.


And here Linux Virtual Server or LVS comes to our rescue. In fact, this is a kernel module ( ipvs ), existing somewhere else from version 2.0-2.2.

What is he like? In fact, this is an L4 router (I would say that L3, but the authors insist on L4), which allows for a transparent and manageable handling of packets along specified routes.

The basic terminology is as follows:

On the director, this same IPVS module (IP Virtual Server) is enabled, the rules for packet forwarding are configured and the VIP is raised - usually as an alias to the external interface. Users will walk through the VIP.

Packages that come to the VIP are forwarded by the selected method to one of the Realservers and are already normally processed there. It seems to the client that he works with one machine.


The rules for packet forwarding are extremely simple: we define a virtual service defined by the VIP: port pair. The service can be TCP or UDP. Here we also set the node rotation method (scheduler, scheduler). Next, we define a set of servers from our farm, as well as a pair of RIP: port, and also specify the packet forwarding method and weight, if required by the selected scheduler.

It looks something like this.

# ipvsadm -A -t -s wlc
# ipvsadm -a -t -r -w 3
# ipvsadm -a -t -r -w 2
# ipvsadm -a -t -r -w 1

Yes, do not forget to put the ipvsadmin package, it should be in the repository of your distribution. In any case, it is in Debian and RedHat.

In the example above, we create a virtual HTTP service and include servers, and in it. The "-w" key sets the server weight. The higher it is, the more likely it will receive a request. If we set the weight to 0, then the server will be excluded from all operations. Very handy if you need to decommission the server.

The default packets are forwarded by the DR method. In general, there are the following options for routing:

DR is the easiest, but if, for example, you need to change the destination port, you will have to draw rules in iptables. NAT, on the other hand, requires the default for the entire farm to be directed to the director, which is not always convenient, especially if real servers also have external addresses.

There are several planners in the delivery package, you can read in detail in the instructions. Consider only the main ones.

In addition, it can be reported that the service requires persistence, i.e. keeping the user on one of the servers for a given period of time - all new requests from the same IP will be transferred to the same server.

Underwater rocks

So, according to the example above, we should have received a virtual server on VIP, which ipvsadm happily reports to us:

ipvsadm -L -n
IP Virtual Server version 1.0.7 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP wlc
-> Route 3 0 0
-> Route 2 0 0
-> Local 1 0 0

However, when trying to connect, nothing will happen! What is the matter? The first step is to raise the alias.
# ifconfig eth0:0 inet netmask

But even here failure is waiting for us - the packet arrives at the real server interface unmodified and therefore stupidly replays the kernel as not intended for the machine. The easiest way to resolve this issue is to raise the VIP to the loopback.

# ifconfig lo:0 inet netmask

In no case do not raise it on the interface, which looks at the same subnet as the external interface of the director. Otherwise, the external router can cache the mac of the wrong machine and all traffic will go wrong.

Now the bags should run where necessary. By the way, the director himself can be a real server, which we see in our example.

Single point of failure

In this decision, the director himself will be the obvious point of failure causing the destruction of the entire service.

Well, it does not matter, ipvsadm supports launching in daemon mode with the ability to synchronize tables and current connections between several directors. One will obviously become a master, the rest will be slaves.

What remains to us? Move VIP between directors in case of failure. Here HA solutions like heartbeat will help us.

Another task will be monitoring and timely decommissioning of servers from our farm. The easiest way to do this is with weights.

For settling both questions a lot of solutions are written, sharpened for different types of services. I personally liked the Ultramonkey solution most of all, from the LVS authors themselves.

RedHat has a native thing called Piranha, it has its own set of daemons for monitoring the farm and directors, and even some kind of ragged web interface, but it does not support more than 2 directors in the HA bundle. I do not know why.


So, Ultramonkey consists of 2 main packages - heartbeat and ldirectord. The first is engaged in providing HA for directors, including raising and moving VIP (and generally speaking can be used not only for directors), and the second maintains the ipvsadm config and monitors the state of the farm.

For heartbeat, you need to draw 3 config. Basic versions are provided with detailed comments, so just give examples.

auth 2
#1 crc
2 sha1 mysecretpass
#3 md5 Hello!

We configure the authorization of demons on different machines.

Here we have information about which resource we are creating HA for and what services we pull when moving.
director1.foo.com IPaddr:: ldirectord

Those. raise on interface eth0 ​​and run ldirectord.

keepalive 1
deadtime 20
udpport 694
udp eth0
node director1.foo.com # <-- uname -n !
node director2.foo.com #

We say how to maintain the HA cluster and who actually belongs to it.

Ldirectord has only one config and there, in general, everything is also clear.
# Virtual Service for HTTP
real= gate
real= gate
receive="I'm alive!"

Those. ldirectord will jerk every 2 seconds through the http file alive.html and if it doesn’t have the line “I'm alive!” or, worse, the server doesn’t respond, the demon will immediately put it weight 0 and it will be excluded from subsequent shipments.

Weights can also be arranged by yourself, for example, by going over the field with a crown and calculating them depending on the current loadavg, etc. - no one takes away direct access to ipvsadm from you.


Although everywhere on the Internet, LVS web server balancing is generally considered as the scope of application, in fact they can balance a great many services. If the protocol is held on one port and has no state, then it can be balanced without any problems. The situation is more complicated with multiport protocols like samba and ftp, but there are also solutions.

By the way, as a real server does not have to be Linux. It can be almost any OS with a TCP / IP stack.

There is also a so-called L7 router, which no longer operates with ports, but with knowledge of high-level protocols. Fellows Japanese are developing Ultramonkey-L7 for this occasion. But now we will not touch him.

What else to read?

What balancing solutions do you use?
I will welcome any comments and comments.

Source: https://habr.com/ru/post/104621/

All Articles