Unix and Linux Initialization Systems after SysV

Until the mid-2000s, it never occurred to anyone to change sysvinit , almost to no one. Gentoo from the very beginning created and developed OpenRC . That all changed with launchd on Mac OS X. Ubuntu developers rushed to create Upstart , in which some ideas from launchd were borrowed. It was a shaky case, but then there was a systemd and mixed all the cards. But who was the true pioneer?

Daniel J. Bernstein is a mathematician and cryptographer, author of the popular MTA qmail and many other lesser-known programs, among which daemontools stands out . For many modern systems of initialization, daemontools was an example and inspiration. I ask you inside to get acquainted with the most elegant, simple and influential service management system in Unix / Linux.

DJB and Daemontools

Maxwell's equations for managing processes on Unix OS .

Attention - do not confuse daemontools written by DJB with the namesake program DAEMON Tools for mounting iso images and creating virtual CD / DVD discs.

Daniel J. Bernstein created his program in 1997. The last stable release was in July 2001.

Now, when there is a split in the Linux community due to systemd, it's time to remember what a real init can be in the spirit of Unix principles and philosophy. In this sense, Zen DJB programmer is the epitome of minimalism and simplicity, and Occam's razor is built into the keyboard. His solutions are solid and elegant, on this foundation he builds reliable, secure software that consumes the minimum amount of OS resources. Here are some features of his work style.

• The largest file source code multilog.c has only 13898, there are no strings, but bytes. The wc command indicates only 617 lines of code.
• Most functions have less than 30 lines.
• The principle is to never parse anything.
• The principle is to use everything that the OS gives and not reinvent the wheel.

Why such strange principles, and even considering the fact that the author professes them fanatically? Daemontools building material - directories, processes, FIFO, executable files. This gives a lot of advantages in the development and debugging of the application:

• Testing service startup is easy - if the ./run executable file ./run , the service also starts.
• You can use any programming language, not just Bash. Even the compiled binary will do.
• It is clear what and how the program does even without a detailed reading of the documentation and studying the source code.
• Parsing a variety of textual structures is a surprisingly difficult task , if you do it according to your mind. The author avoids this by skillfully using the hierarchical property of the Unix file system to recreate the structure of environment variables = .

DT comparison table

It is worth paying attention to the unorthodox structure of the daemontools directories, in no case the program creates directories in the root of the Unix file system. DJB has specified the /service , /command directories in the program code and recommends creating /package for the program sources. This is considered to be a very bad form in Unix and Linux, the creators of distributions avoid this by all means, as well as root users.

Let's go over the daemontools self-praise table . Let's start with the first line. Indeed, creating and deleting a new service is easier than ever, adding or deleting a new directory in /service along with the ./run file and that's it. Compare with sysvinit and other init scripts to appreciate the simplicity of such a way to achieve the same.

The second point is less convincing in the sense that, of course, it is easier when the service starts automatically for the first time, but in the rest of the systems for initializing and managing services, one command is enough for the first start of the service.

The ability to restart the completed service in automatic mode was at that time a great step forward. Today it is in the order of things.

The fourth position is signals. The svc command, as shown below, allows you to send almost any signal of POSIX standards to the service.

The last two positions seem somewhat contrived, it is not entirely clear how daemontools restores the state of the process, unlike the rest of the ones. It is also not clear why the author only considers his program portable to other platforms.

DT structure

According to the author: daemontools is a set of tools for managing UNIX services. The main differences from traditional inits (directory structures rcX.d, rc.d, rc.local, etc.) is the ability to restart the service if it fails and the presence of a logging and rotation program - multilog. Also, multilog allows you to log the output of programs that can not redirect the output to syslog . Thus, you can run as a service program, for this not intended.

The internal structure of the daemontools is outlined by a red dashed line.

Now a little about the principles of the program.

At the very beginning, the system init starts svscanboot , which then runs the svscan program in the newly created svscan directory. Next, svscanboot redirects the output of the running svscan to the svscan debugging process.

The core of daemontools are just two programs: svscan and supervise . The first one starts with the only optional argument, and the second one with the required key.

Svscan is used to launch and track services. Every 5 seconds, Svscan checks the /service directory, if not specified, for new subdirectories. If these are found, a new copy of supervise launched for each directory.

Supervise is, according to the name, controlling the process. It is called with a parameter that contains the name of the directory and looks for the ./run script in ./run , which starts it. If for some reason ./run stopped executing, then supervise restart it after a short pause - so as not to create an additional load on the OS. Supervise will not restart ./run if ./run is found in the directory. Supervise creates a ./supervise subdirectory in the service directory that stores process data. This data can be read using the svstat utility. To control the service is the program svc .

The svc command syntax and options are shown below.

 svc options services 

• -u: Up, start the service, restart if stopped.
• -d: Down, stop the service.
• -t: Terminate, sends the TERM signal to the service.
• -k: Kill, sends a KILL signal to the service.
• -p: Pause, sends the STOP signal to the service.
• -c: Continue, sends the CONT signal to the service.
• -h: Hangup, sends a HUP signal to the service.
• -a: Alarm, sends the ALRM signal to the service.
• -h: Interrupt, sends an INT signal to the service.
• -x: Exit, supervise will exit as soon as ./run or its descendant is completed.
• -o: Once, start the service, but do not restart after it is completed.

There is another circumstance, if the working directory supervise contains a subdirectory ./log , in which there is ./log/run , then another copy of supervise will be launched and a channel created between ./run and ./log/run .

Let's try to add the sshd service.

 #!/bin/sh #  stderr  stdout exec 2>&1 #   -D sshd  ,   -e    stderr exec /usr/sbin/sshd -D -e 

In this case, the directory structure may look like this.

 - service/ |- ngetty/ | |- run | |- log/ | |- run |- sshd/ | |- run | |- log/ | |- run |- squid/ | |- run | |- log/ | |- run 

After svscan runs through this list, we get a process tree, in which the service processes monitor services and logging.

 -svscan-+-service-+-ngetty | `-log-service +-service-+-sshd | `-log-service +-service-+-crond | `-log-service 

Dependencies between services

Despite the fact that the program does not support dependencies between different services, there is a way to achieve dependency accounting using svok as follows.

 #!/bin/sh svok postgres || (echo "waiting for postgres..." && exit 1) exec 2>&1 exec python3 your-web-app.py 

• svok - checks if a particular service is running. Nothing output to stdout and stderr , if the service is running, then the program returns code 0, otherwise code 100.

In this example, the python program will start only if postgres has started, but if the latter has not yet risen, the script will end and then after a certain time svscan restarts it. When postgres finally comes up, python will launch the web application.

Quoting service

Using the softlimit utility, softlimit can limit the resources provided to this service.

 #!/bin/sh exec 2>&1 #    foo,   4096 ,  #   15    1: exec setuidgid foo softlimit -n 4096 -o 15 -p 1 \ bar -n 

• softlimit - runs the program with resource constraints.

Logging

If you have a certain program foo , which does not keep logs, you can easily do it with the help of multilog , collecting in a separate file the output of stdout and stderr with timestamps.

 [root@home: +5] cd /service/foo && mkdir log [root@home: +5] cd log && mkdir main [root@home: +5] nano run.new #   #!/bin/sh exec 2>&1 exec multilog t ./main [root@home: +5] chmod u+x run.new [root@home: +5] mv run.new run [root@home: +5] cd /service [root@home: +5] svc -t foo 

From another terminal, run:

 tail -fn0 /service/foo/main/current 

Daemontools followers

Few people use DT today, but we can safely say that Daniel J. Bernstein has become an example for many, and his work is alive and well. Here is a partial list of his followers.

• daemontools-encore - Developer Bruce Guenter, is a further development of DT. Not a full-fledged init, as well as the original.
• runit - Developer Gerrit Pape, able to run services in parallel. Habrapost .
• s6 - Full asynchronous init with PID 1.
• nosh - Lightweight process hypervisor for BSD and Linux, able to simultaneously start and stop services.

Used materials

1. Celebrating daemontools
2. Daemontools Intro
3. Introduction to Daemontools
4. A history of modern init systems (1992-2015)