How it all began

In the course of working with microservices, we have repeatedly encountered problems with the discovery service during autoskelling, collapsing extra nodes.

Almost all solutions that were existing or existing at the moment were tried, but as usual - nothing fit perfectly on our dynamic environments (dozens of stops / launches of containers of the same type per hour). The closest solution was NGINX + Consul + Consul templates, but it was ugly, required a restart, did not allow using external headers except through Consul.

In general, as always happens - it was decided to write his decision. In the course of the discussion, dozens of things surfaced that would be good to implement, of which the most critical for us and interesting to the public were chosen.

Pledged functionality on the first iteration

Types of balancing:

Iphash
Leastconn
Roundrobin
Weight

Discovery (definition of a pool of backends for each frontend):

Static - just a list of servers in the config.
Docker - request to the Docker / Swarm API docker filtered by label and internal ports of the container.
Exec - initiating the launch of an external script, reading from stdout and parsing it regularly (as long as it is spelled out strictly).
JSON - requests a URL via an HTTP request and parses it by patterns (supports multi-level JSON).
Plaintext - requests a URL via an HTTP request and parses it according to the regular expression specified in the config.
SRV — Requests a DNS SRV record for the service name.

Helscheki

Since it was initially quite clear that we simply would not pull out to implement the helschacks in the same haproxy volume, it was decided to get rid of a little blood and make only 2 types of helschacks.

Ping - simple TCP ping;
Exec - launching an arbitrary binary with passing parameters to it and reading output from stdout.

Examples of using

SRV Discovery

We have arbitrary services that register themselves, for example, in Consul. We will use Consul dns to define a pool of servers.

In this example, we defined the type of balancing as "srv", the DNS server and its port are also defined, to which requests for the service of discovery will go. The frequency of updating the server list was determined, as well as an important variable - the policy for the case when the DNS server did not respond. For maximum consistency of the environment, you need to set failpolicy = "setempty". In this case, if there is no response from DNS, the entire pool of backend servers will be reset, and incoming connections will be dropped. Otherwise, you need to use failpolicy = "keeplast", then the balancer will use the latest data that came before the DNS connection failed.

toml [servers.sample2] bind = "localhost:3001" protocol = "tcp" balance = "weight" [servers.sample2.discovery] failpolicy = "keeplast" kind = "srv" srv_lookup_server = "66.66.66.66:8600" # dns server and port srv_lookup_pattern = "api.service.ireland.consul." # SRV service pattern [servers.sample2.healthcheck] fails = 1 passes = 1 interval = "2s" kind = "ping" timeout = "500ms" 

Docker / Swarm Balancing.

In essence, there are no differences in the API and the configuration method for the Docker / Docker Swarm. We can work equally well with the Docker host and with the Docker Swarm cluster. Consider working with them in one example.

Now we will balance certain services using Docker Swarm as a more general example. Everything described below works for a separate Docker host. In this example, we define the discovery type as "docker", define the base docker url, labels and internal port of the container docker (from the network of the container itself) by which the balancer will select the pool from which backend servers are formed.

For this example, we will use a more advanced type of helschack, namely exec-helschack. In addition to the parameters of the startup check frequency, there is also a time to work out the script. The time between launches should be longer than the time to work out the script so that there are no “raids”. The command to start this helcheck is formed as / path / to / script [ip] [port]. After working out the script, it should output to stdout a string that is compared with a positive and negative expected result.

 [servers.sample3] bind = "localhost:3002" protocol = "tcp" balance = "weight" [servers.sample3.discovery] interval = "10s" timeout = "2s" kind = "docker" docker_endpoint = "http://localhost:2377" # Docker / Swarm API docker_container_label = "api=true" # label to filter containers docker_container_private_port = 80 # gobetween will take public container port for this private port [servers.sample3.healthcheck] kind = "exec" interval = "2s" exec_command = "/etc/gobetween/checks/exec_healthcheck.sh" exec_expected_positive_output = "1" exec_expected_negative_output = "0" exec_timeout_duration = "1s" 

In subsequent articles, I plan to give a few examples of more complex use of other types of discovery. Specific configuration and installation under Windows will also be described.

• A closer example to life
• Project on Github
• Wiki on github
• performance tests
• Installation under Windows
• Continued.Gobetween Exec discovery + Elasticsearch