Daemon Decoration or Gearman Infrastructure

Gearman - a framework for the distribution of tasks. You can read more about it: Here or Here . This tool is a find for solving a variety of tasks. But during its operation on combat servers, we encountered some problems, the solution of which I want to share:
- How to automatically start the demons and monitor their health?
- How to run multiple instances of workers?
“Where will the demons write messages to be displayed on the screen?”
- And how to make sure that the demons are not distracted by busty gargoyles , and worked. word follow the load.
There are quite convenient tools for this:


We assume that we have a running and configured Gearman-server, workers and clients who periodically want something from workers.
')

Supervisor

Problem: We started the Gearman server, now we need to start the n-th number of different workers.
- Run manually and untie from the console?
- and how to find out that someone fell? and is the work going?
- ah, do not untie from the console. No, it smacks of masochism.

Supervisor can be used to solve these problems.

As stated in the manual: Supervisor is a client-server system that helps users monitor and manage the number of processes in the Unix-like operating system.
With it, we can demonize all workers in the right quantity, as well as track their work.
It is necessary for each worker to write a separate config file and put it in /etc/supervisor/conf.d/
The config file has a fairly simple structure:

[program:<worker_name>] command=<execution_file> process_name=%(process_num)s stdout_logfile=/var/log/<worker_name>.log stderr_logfile=/var/log/<worker_name>.log redirect_stderr=true autostart=true autorestart=true startsecs=5 numprocs=1 

[program: <worker_name>] - section header. The following parameters will apply only to this worker (daemon)
command - the command to start the file. most often the path to the executable file.
process_name is the name of each worker instance. There is a subtlety - The manual indicates that this parameter is optional. But I had problems running a few worker instances, if not specified. The fact is that the process names must be different. Here, in addition to the name of the worker, we indicate its sequence number.
stdout_logfile - speaks for itself. where we redirect the standard output console.
stderr_logfile is the same, only for errors. I recommend to specify these 2 parameters. They can really help out when they have to figure out why your demons are falling.
autostart - as expected. we start vorker at start of the supervisor. Typically, when the system starts.
autorestart - already for one this parameter it is worth using a supervisor. Restart your worker if he fell with an Exception while performing one of the tasks.
startsecs - how long does a worker need to wait for him to start (optional parameter)
Numprocs is the second parameter for which you can love the supervisor - the number of worker instances.

Control and management of demons

There are 2 ways to control daemons that are started by the supervisor: this is a console utility, supervisorCtl, and a web interface.
SupervisorCtl at startup displays information about the status of daemons:

 Gearman_AddressCodeFinder:0 RUNNING pid 2060, uptime 10:15:01 Gearman_MessageSender:0 RUNNING pid 2064, uptime 10:15:01 Gearman_MessageSender:1 RUNNING pid 2063, uptime 10:15:01 Gearman_ReportCreator:0 RUNNING pid 2055, uptime 10:15:01 Gearman_WorkplanNotifier:0 RUNNING pid 2061, uptime 10:15:01 Universal:0 RUNNING pid 2059, uptime 10:15:01 Universal:1 RUNNING pid 2058, uptime 10:15:01 

Here everything is clear and without explanation. You can understand who works, who fell.
Using this utility from the console, you can stop / start daemons, reconfigure the supervisor. Quite interesting is the fg command, which allows you to connect to the process in the background (it becomes attached to your console). Pretty handy thing for debugging on the combat system :)

Manage daemons from the web interface

In addition to the console interface, the Supervisor has a web interface:


You can usually get into it by Url: 127.0.0.1 : 9211 / In your case, the port may differ. settings can be found in the section

 [inet_http_server] port = 127.0.0.1:9211 

file config (in my case /etc/supervisor/supervisord.conf)

From the web interface, as well as from the console, you can start / stop daemons, make sure that they are output to the console. The latter is especially valuable when you have to catch demons' bugs on the combat system.

Gearman tools

Monitoring demons is good. But how do you know how many tasks are on the queue server, and how many and what tasks are being performed at the moment. There are at least 2 ways:
1) Connect via telnet to the queue server and request status:

 telnet localhost 4730 status //     address:find 0 0 1 workplan:sendnotify 0 0 1 report:createreport 0 0 1 message:send 0 0 2 universal:universal 0 0 2 

But ifnnormatsiya pretty scanty. and will you remember in 2 months what is the number in the 3rd column? I am definitely not.
2) There is a more beautiful solution - gearman_top utility
It shows the same information, but in a more convenient form, and the data is updated in real time.
Installing is quite simple:

 apt-get install gearman-tools 

Run by the command:

 gearman_top 

Displays information:

 Queue Name | Woker Available | Jobs Waiting | Jobs Running ---------------------------------------------------------------------- address:find | 1 | 0 | 0 message:send | 2 | 0 | 0 report:createreport | 1 | 0 | 0 universal:universal | 2 | 0 | 0 workplan:sendnotify | 1 | 0 | 0 ---------------------------------------------------------------------- 

The text of the output on the screen speaks for itself: Which queues are present on the server, how many workers are available at the moment, how many tasks are being performed and how many are waiting to be completed.