Restart daemon in PHP without losing connections to it

At various conferences, we repeatedly told about our cloud for CLI scripts ( videotape of the report , slides ). The cloud is designed to run various PHP scripts on a schedule or through an API. As a rule, these scripts process queues, and the load is spread over approximately 100 servers. Previously, we focused on how the control logic is implemented, which is responsible for evenly distributing the load over such a number of servers and generating tasks on a schedule. But beyond that, we needed to write a daemon that would be able to run our PHP scripts in the CLI and monitor their execution status.

It was originally written in C, like all the other demons in our company. However, we are faced with the fact that a significant part of the processor time (about 10%) was spent, in fact, wasted: it is the launch of the interpreter and the loading of the “core” of our framework. Therefore, in order to be able to initialize the interpreter and our framework only once, it was decided to rewrite the daemon in PHP. We called it Php rock syd (by analogy with Phproxyd - PHP Proxy Daemon, the C demon that we had before). It accepts requests to launch individual classes and does fork () for each request, and is also able to report on the execution status of each of the launches. This architecture is in many ways similar to the Apache web server model, when all initialization is done once in the “wizard” and the “children” are already engaged in processing the request. As an additional “bun”, we get the ability to enable opcode cache in the CLI, which will work correctly, since all children inherit the same area of ​​shared memory as the master process. To reduce delays in processing the launch request, you can fork () in advance (prefork-model), but in our case, fork () delays are about 1 ms, which is fine with us.

However, since we update the code quite often, this daemon also has to be restarted frequently, otherwise the code that is loaded into it may become outdated. Since each restart would be accompanied by a lot of connection reset by peer errors, including end-user service failures (the daemon is useful not only for the cloud, but also for part of our site), we decided to look for ways to restart the daemon without losing already established connections. There is one popular technique that makes a graceful reload for daemons: fork-exec is done and the descriptor from the listen-socket is passed to the child. Thus, the new connections are already accepted by the new version of the daemon, and the old ones are being modified using the old version.

In this article we will look at the complicated version of graceful reload : the old connections will continue to be processed by the new version of the daemon, which is important in our case, because otherwise it will run the old code.
')

Theory

Let's first think: is it possible that we want to get? And if so, how can this be achieved?

Since the daemon runs on Linux, which is POSIX-compatible, the following features are available to us:

1. All open files and sockets are numbers corresponding to the number of the open descriptor. Standard input, output, and error stream have descriptors 0, 1, and 2, respectively.
2. There are no significant differences between the open file, the socket and the pipe (pipe) (for example, you can work with sockets using both read / write and sendto / recvfrom system calls).
3. When executing the fork () system call, all open descriptors are inherited with their numbers and read / write positions (in files) preserved.
4. When execve () system call is executed, all open descriptors are also inherited, and in addition the PID of the process is saved and, therefore, binding to its children.
5. The list of open process descriptors is available from the / dev / fd directory, which in Linux is a symlink on / proc / self / fd.

Thus, we have every reason to believe that our task is accomplished, and without much effort. So let's get started.

PHP patches

Unfortunately, there is one small detail that complicates our work: in PHP there is no possibility to get the file descriptor number for streams (streams) and open the file descriptor by number (instead, a copy of the file descriptor opens, which is not suitable for our daemon, since we are very we carefully monitor open handles in order not to create leaks when restarting and when starting child processes).

To begin, we will make a couple of small patches in the PHP code to add the ability to get fd from the stream (stream) and make fopen (php: // fd / <num>) not open the copy of the descriptor (the second change is incompatible with the current PHP behavior, so you can add a new “address” instead, for example, php: // fdraw / <num>):


We added the fd field to the result returned by the stream_get_meta_data () function if it makes sense (for example, the fd field will not be present for zlib streams). We also replaced the dup () call from the transferred file descriptor with a simple check. Unfortunately, this code will not work without modifications under Windows, since the fcntl () call is POSIX-specific, so the full patch must contain additional code branches for other OSs.

The demon without the possibility of restarting

To begin with, we will write a small server that can receive requests in JSON format and give some answer. For example, it will give the number of elements in the array that came in the request.

The daemon is listening on port 31337. The output should be something like:

 $ telnet localhost 31337 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. {"hash":1} #   "Request had 1 keys" {"hash":1,"cnt":2} #   "Request had 2 keys" 

We will use stream_socket_server () to start listening to the port, and stream_select () to determine which descriptors are ready to read / write.


The code for this daemon is more than standard, but I would like to note one implementation detail: we store all the read and write buffers with binding to specific connections and perform request processing right in the same place where we read the request. This is important because one of such requests may be restart, and in this case it will not come to processing the following requests. However, since we have not yet read the requests, the next time stream_select () from the same descriptors will return the same result. Thus, we will not lose a single request if we restart directly from the command handler (except when several commands are sent to us at the same connection, and restart is one of these commands).

So, how to make it possible to restart the daemon?

Daemon with restarting and maintaining established connections

Our simplest example did not know how to do anything useful, so let's still write the demon that was discussed at the very beginning. We want to get something like the following (commands to the daemon are sent as “command_name [JSON-data]”, response as JSON):

 $ telnet localhost 31337 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. #      restart #      "Restarted successfully" #    run {"hash":1,"params":[1,2,3],"class":"TestClass1"} #   {"error_text":"OK"} #     ( child TestClass1   ) restart "Restarted successfully" #   :    check {"hash":1} {"error_text":"Still running"} #  5     :  TestClass1   check {"hash":1} {"retcode":0} #     ,    free check {"hash":1} {"retcode":0} free {"hash":1} {"error_text":"OK"} restart "Restarted successfully" #   ,          restart restart {"error_text":"Restarted successfully"} bye Connection closed by foreign host. 

The idea for restarting is simple: we will create a file with all the necessary information, and when we start, we will try to read it and restore the open file descriptors.

First, we write the code to write to the restart file:

 echo "Creating restart file...\n"; if (!$res = $this->getFdRestartData()) { fwrite(STDERR, "Could not get restart FD data, exiting, graceful restart is not supported\n"); exit(0); } /* Close all extra file descriptors that we do not know of, including opendir() descriptor :) */ $dh = opendir("/proc/self/fd"); $fds = []; while (false !== ($file = readdir($dh))) { if ($file[0] === '.') continue; $fds[] = $file; } foreach ($fds as $fd) { if (!isset($this->known_fds[$fd])) { fclose(fopen("php://fd/" . $fd, 'r+')); } } $contents = serialize($res); if (file_put_contents(self::RESTART_DIR . self::RESTART_FILENAME, $contents) !== strlen($contents)) { fwrite(STDERR, "Could not fully write restart file\n"); unlink(self::RESTART_DIR . self::RESTART_FILENAME); } 

The code for getting an array of data (the getFdRestartData () function) is shown below:

 $res = []; foreach (self::$restart_fd_resources as $prop) { $res[$prop] = []; foreach ($this->$prop as $k => $v) { $meta = stream_get_meta_data($v); if (!isset($meta['fd'])) { fwrite(STDERR, "No fd in stream metadata for resource $v (key $k in $prop), got " . var_export($meta, true) . "\n"); return false; } $res[$prop][$k] = $meta['fd']; $this->known_fds[$meta['fd']] = true; } } foreach (self::$restart_fd_props as $prop) { $res[$prop] = $this->$prop; } return $res; 

The code takes into account that we have 2 types of properties:

1. Properties containing resources with connections: $ restart_fd_resources = ['read', 'write', 'streams'].
2. Properties containing buffers and other information about connections that can be “serialized” in raw form: $ restart_fd_props = ['read_buf', 'write_buf', 'conn_count'].

We also remember all the fd stored in the restart file, and close all the others (if any), because otherwise you can prevent the leaking of file descriptors.

Next we need to load this file at the start and continue to use open handles as if nothing happened :). The code for the two functions (loading a restart file and loading file descriptor information) is as follows:

File upload:

 if (!file_exists(self::RESTART_DIR . self::RESTART_FILENAME)) { return; } echo "Restart file found, trying to adopt it\n"; $contents = file_get_contents(self::RESTART_DIR . self::RESTART_FILENAME); unlink(self::RESTART_DIR . self::RESTART_FILENAME); if ($contents === false) { fwrite(STDERR, "Could not read restart file\n"); return; } $res = unserialize($contents); if (!$res) { fwrite(STDERR, "Could not unserialize restart file contents"); return; } foreach (self::$restart_props as $prop) { if (!array_key_exists($prop, $res)) { fwrite(STDERR, "No property $prop in restart file\n"); continue; } $this->$prop = $res[$prop]; } $this->loadFdRestartData($res); 

The loadFdRestartData () function to expand the file descriptor array back:

 $fd_resources = []; foreach (self::$restart_fd_resources as $prop) { if (!isset($res[$prop])) { fwrite(STDERR, "Property '$prop' is not present in restart fd resources\n"); continue; } $pp = []; foreach ($res[$prop] as $k => $v) { if (isset($fd_resources[$v])) { $pp[$k] = $fd_resources[$v]; } else { $fp = fopen("php://fd/" . $v, 'r+'); if (!$fp) { fwrite(STDERR, "Failed to open fd = $v, exiting\n"); exit(1); } stream_set_read_buffer($fp, 0); stream_set_write_buffer($fp, 0); stream_set_blocking($fp, 0); stream_set_timeout($fp, self::CONN_TIMEOUT); $fd_resources[$v] = $fp; $pp[$k] = $fp; } } $this->$prop = $pp; } foreach (self::$restart_fd_props as $prop) { if (!isset($res[$prop])) { fwrite(STDERR, "Property '$prop' is not present in restart fd properties\n"); continue; } $this->$prop = $res[$prop]; } 

We re-set the read_buffer and write_buffer values ​​for open file descriptors and set timeouts. Oddly enough, after these manipulations, PHP calmly does accept () on these file descriptors and continues to read / write normally in them even though it does not know that these are sockets.

In the end, we need to write logic to launch and monitor the execution status of the workers. Since this is not related to the topic of the article, the full implementation of the daemon is placed on the github repository, the link to which is given below.

Conclusion

So, this article described the implementation of a daemon that communicates using the JSON protocol and is able to run arbitrary classes in separate processes while tracking the process of their execution. To launch individual classes, the fork () model is used for the request , therefore, to process the request, the interpreter does not need to be restarted and the framework is loaded, and it becomes possible to use opcode cache in the CLI. Since every time the code is updated, the daemon needs to be restarted, it is necessary to provide a mechanism for smoothly restarting this daemon (in our company, the code is sometimes updated every few minutes, in the form of “hotfixes”).

The restart occurs by execve () execve () system call, as a result of which all descendants remain attached to the parent (since the process PID does not change during execve ()). All open file descriptors are also saved, which allows you to continue processing requests from users in already open connections. All network buffers, information about running children and open descriptors are stored in a separate restart file, which is read by a new daemon instance, after which work continues in the standard event loop.

The full implementation code can be seen on GitHub at the following address: github.com/badoo/habr/tree/master/phprocksyd

Questions, suggestions, clarifications are welcome.

Respectfully,
Yuri youROCK Nasretdinov
Lead PHP developer
Badoo