All happiness and good!

Evolutionarily it turned out that in my personal possession there was not a small zoo of various servers: from cheap Supermicro to top-end (at the time of release) HP Gen 8. Everything is of course connected by optics and other pleasures of life.

But the tale is not about how to put the network, and not even about how to configure the server, but about how right just on all this, docker-compose services should be raised and rejoiced.

It would seem that it could be simpler: he took, made docker-compose.yaml, a couple of Dockerfile, launched - profit. But the demon is in the details ..

The prelude about the server is given for a reason: what if there are ten or two virtual machines in each server, and hundreds of services out of ten compose files in each virtual machine? In the appendage, we must not forget that part of the components simply exist as independent services.

And everything seems to work. But sometimes that marvelous wonderful moment comes when a consumer of one of the services comes up with a question of the type “but something I’ve already had a week’s service is so little not alive”.
And it seems like the client is loyal (the service is free for him), and it seems that the tragedy did not happen, but a strange sensation of an impending storm of unexpected people and calls to three nights appears.

Since the most important skill of any programmer / devopser / admin is the ability to respond in time to signals from the softest part of the body just below the back, I began to take appropriate measures.

But I would not be myself and not a mathematical engineer, if I had not tried to formulate the causes and possible solutions to the problem.

Given

• Zoo of services running both in and without the docker
• The number of services will increase
• Some groups of services are completely isolated (usually in order to distinguish visibility / access)

Task

• Make it so that nothing falls. To rise itself in the fall
• Use solutions that are possible for a small number of services, and for large
• Know when and what fell. Preferably no later than the client.
• Do not spend a lot of time setting

Decision

First, the temptation to start services with restart = always was marked by convulsive memories of loaded containers with zero delay at the start.
Secondly, smoking docker-compose v3 showed that something like this is there, but on close inspection it works with a lot of conventions.
Third, the use of Kubernetus and co., Looked like a shot from the SRZO Smerch in a mosquito at a distance of 100 meters.

After a couple of strong mugs Roma coffee, the memory of Consul and the consul-template came to mind.

The following chain was formed:

• consul-template runs the application
• It is registered as a service in the Consul (Service Register),
• and I fasten monitoring on the same consul due to events and change monitoring

Rainbows, unicorns and butterflies ... Until I realized that the consul-template cannot register with Consul. 0_o
link to the Issue , which leads to the Issue 2016

However, the idea was still simple:

Dear Microsoft founder, bequeathed us all to be lazy, but since who I am to argue with him, I went to look for alternative solutions.

Suddenly it turned out that there was no easy solution. Supervisord - pulls a python (I don’t have anything against it, but it's kind of a pity, especially considering the number of services). Zabbix and others like him are good, but either heavy agents or a big delay.

# izheprogrammer so for cooking. The recipe is simple: let's take go (the final files are small and self-sufficient, and the cross-compilation is elementary), a handful of knowledge on unix-processes and a lot of tea.

Chronology (D - day start):

• D + 1 ready, working prototype
• D + 2 fixed all found bugs
• D + 3 given at the mercy of colleagues
• D + 4 errors found by colleagues
• D + 7 everything was rewritten
  ...

Our days

An ultra-lightweight supervisor was made with the ability to configure the number of restarts, the intervals between them, automatic (de) registration with Consul, notification with Telegram.
Startup parameters can be saved to a configuration file. The root part is rendered as a separate library, the ability to easily develop extensions has been added.

Example

Suppose there is a service (let's call it / usr / local / bin / dummy) with a large number of flags to configure. He can sometimes fall (he is not, life is =).

We start it with registration in the local Consul agent:

monexec run --consul -- /usr/local/bin/dummy -a 1 -b 2 -c 3 --foo bar --foo baz --foo etc 

Profit! By default, monexec will restart it forever with an interval of 5 seconds.

Next, we want to transfer this to administrators for future use. Kilometer start line to send somehow not comme il faut.

We generate the startup script (all the same, just add the --generate flag):

  monexec run --generate --consul -- /usr/local/bin/dummy -a 1 -b 2 -c 3 --foo bar --foo baz --foo etc > dummy.yml 

 services: - label: stinger-desert command: /usr/local/bin/dummy args: - -a - "1" - -b - "2" - -c - "3" - --foo - bar - --foo - baz - --foo - etc stop_timeout: 5s restart_delay: 5s restart: -1 consul: url: http://localhost:8500 ttl: 3s timeout: 1m0s register: - stinger-desert 

This will create a configuration file for us to use.
Run it like this:

  monexec start /path/to/dir/with/dummy 

Important! /path/to/dir/with/dummy is the directory, not the path to the file. (Inspired by Dockerfiles)

Now add a happy life and a gram of madness: connect alerts in Telegram. To do this, create a bot and find out your number.

 services: - label: stinger-desert command: /usr/local/bin/dummy args: - -a - "1" - -b - "2" - -c - "3" - --foo - bar - --foo - baz - --foo - etc stop_timeout: 5s restart_delay: 5s restart: -1 consul: url: http://localhost:8500 ttl: 3s timeout: 1m0s register: - stinger-desert telegram: token: "123456789:AAAAAAAAAAAAAAAAAAAAAA_BBBBBBBBBBBB" services: - "stinger-desert" recipients: - 123456789 template: | *{{.label}}* Service {{.label}} {{.action}} {{if .error}}️ *Error:* {{.error}}{{end}} _time: {{.time}}_ _host: {{.hostname}}_ 

Total

As one person said, the creation of something new consists of three stages:

1. Idea
2. Implementation
3. Publication

Good or bad idea and implementation (it's up to you) - all this has no value if others cannot use it.

I tried to create a documented source code , a guide to using and compiling binary files for most platforms.

Good luck to all!