Load Balancers for Orchestration Systems

There are more requirements for Load Balancers in orchestration systems (Kubernetes, Nomad and others) than just load balancing. First, Load Balancer should be able to read a directory with a list of services to which traffic needs to be redirected (or, alternatively, enable services to register for their inclusion in traffic). Second, do it dynamically, because Orchestration systems at any time can increase or decrease the number of service replicas, or move them to other addresses on the network. And, thirdly, do it without stopping traffic.

In today's post, I will describe the work with two Load Balancers - Traefik and HAProxy. These Load Balancers have the opportunity to work with an impressive list of orchestration tools. The examples will describe how to work with the Nomad orchestration system.

In the previous post, I already gave an example of Load Balancers - Fabio. Its limitations: it works only with http / https protocols, and works only with Consul. Unlike Fabio, Load Balancers Troefik works with an impressive number of different systems. Here is a partial list taken from the developer’s website: Docker, Swarm mode, Kubernetes, Marathon, Consul, Etcd, Rancher, Amazon ECS, ...

I will continue the example from the previous message , in which several replicas of the django service were created.
')
Traefik can be downloaded from the developer's site as an executable file for the most common operating systems. To integrate with Nomad (actually with Consul) you need to create a configuration file:

[entryPoints] [entryPoints.http] address = ":5001" [web] address = ":8080" [consulCatalog] endpoint = "127.0.0.1:8500" domain = "consul.localhost" exposedByDefault = false prefix = "traefik" 

And then run the command with the following configuration file:

 traefik -c nomad/traefik.toml 

After that, UI Traefik will be available on port 8080, in which no service has been published yet. There are several ways to publish services that ultimately do the same thing - they load key / value data into the Traefik system. We will use the opportunity to set key / value pairs through the tags of services. Let's add the tags parameter to the django service configuration file:

 job "django-job" { datacenters = ["dc1"] type = "service" group "django-group" { count = 3 restart { attempts = 2 interval = "30m" delay = "15s" mode = "fail" } ephemeral_disk { size = 300 } task "django-job" { driver = "docker" config { image = "apapacy/tut-django:1.0.1" port_map { lb = 8000 } } resources { network { mbits = 10 port "lb" {} } } service { name = "django" tags = [ "traefik.enable=true", "traefik.frontend.entryPoints=http", "traefik.frontend.rule=Host:localhost;PathStrip:/test", "traefik.tags=exposed" ] port = "lb" check { name = "alive" type = "http" path = "/" interval = "10s" timeout = "2s" } } } } } 

In this example, the service will be published on the host localhost and mounted to the route / test. Troefik has developed a flexible and complete system of rules for configuring routes, including working with regular expressions. The list of parameters for the rules in the developer documentation.

After executing the nomad job run nomad/django.conf rules will be applied, and traffic from the Load Balancer will be sent to the service. Accordingly, you can change these parameters, deploy the new version of the service with the nomad job run nomad/django.conf , and all changes will be applied without annoying traffic stoppage.

The disadvantage of Troefik is that it works with the protocols of the http / https family (just in case, I note that this family also includes web sockets). But there is still the possibility that it will be necessary to work with other protocols. Therefore, we proceed to the next broader solution based on HAProxy. Some time ago, HAProxy had problems with “soft” overload, which made its use with orchestration systems difficult (during the restart it was necessary to suspend the movement of packets at the network level). Now it is no longer a problem.

First you need to install haproxy on your computer. It does not fit the option with the installation inside the container. In haproxy, only recently there was an opportunity to restart the process in the “soft” mode, but the container still stops the docker, since the second process is actually started with haproxy, just changing them takes place in standby mode - which does not work with the docker and his “one container one process.

For haproxy to work, you must have a configuration file that contains the necessary rules. Nomad (Consul itself) uses a template system that can generate configurations:

 global debug defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 frontend http_front bind *:5001 stats uri /haproxy?stats default_backend http_back backend http_back balance roundrobin{{range service "django"}} server {{.Node}} {{.Address}}:{{.Port}} check{{end}} 

The range keyword in this case plays the role of an iterator. For the three django services, the following configuration file will be generated:

 global debug defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 frontend http_front bind *:5001 stats uri /haproxy?stats default_backend http_back backend http_back balance roundrobin server 228.195.86.224 127.0.0.1:21469 check server 228.195.86.224 127.0.0.1:25872 check server 228.195.86.224 127.0.0.1:25865 check 

To start the generation process using the on-the-fly template, the https://github.com/hashicorp/consul-template library is used. From the developer's resource, you can download the executable file for all common operating systems, and start the process on behalf of a non-privileged user with the command:

 consul-template -template="haproxy/haproxy.cfg.tmpl:haproxy/haproxy.cfg:./haproxy/haproxy.reload.sh" 

The -template parameter contains colon-separated parameters 1) the name of the template, 2) the name of the configuration file to be received 3) a command that is executed after the file is generated. The file will be automatically generated if the variables included in the template are changed (for example, the number of replicas of the django service is changed).

After starting the template engine that generates the first configuration, you can run haproxy:

 haproxy -D -f haproxy/haproxy.cfg -p `pwd`/haproxy.pid 

We explicitly specify the pid-file in order to be able to send a signal to the “soft” haproxy overload:

 haproxy -D -f ./haproxy/haproxy.cfg -p `pwd`/haproxy.pid -sf $(cat `pwd`/haproxy.pid) 

In this example, the service is published on port 5001. At the same port, you can view the statistics of haproxy itself at /haproxy?stats .

UPDATED 02.24.2019 10:43

According to the @usego commentary, additional work has been clarified on the work of haproxy in the docker container, in particular, on the fragment from the github.com/neo4j/docker-library-docs/tree/master/haproxy#reloading-config documentation

Reloading config

If you’ve been using it, you’ll be able to use it.

$ docker kill -s HUP my-running-haproxy

This is an example of the hack-up hacky case and the hack-up case. It is possible to make it possible to use it.

With this approach, the configuration does reload, but as a result of the interruption of the current process. This means that the services will have a very small, but still unavailable period and some customers may see an error message. But sometimes this is not the main selection criterion. Therefore, I’ll give in addition the docker-compose.yml configuration for running haproxy in the docker:

 version: '3' services: haproxy_lb: image: haproxy volumes: - ./haproxy:/usr/local/etc/haproxy network_mode: host 

The command that will overload the haproxy configuration will also change:

 consul-template -template="haproxy/haproxy.cfg.tmpl:haproxy/haproxy.cfg:docker kill -s HUP $(docker-compose ps -q haproxy_lb)" 

The advantages of this implementation include the ability to work without installing haproxy.

Sample code is available in the repository.

Useful links:

1. www.haproxy.com/blog/haproxy-and-consul-with-dns-for-service-discovery
2. m.mattmclaugh.com/traefik-and-consul-catalog-example-2c33fc1480c0
3. www.hashicorp.com/blog/load-balancing-strategies-for-consul

apapacy@gmail.com
February 24, 2019