Infect for the good: how do we execute the spurious code

Recently we have been talking a lot about CRIU - the system of live migration of containers. But today we will talk about an even more interesting development: live application patching, as well as the Compel library, which allows you to do all these outrages, giving hyperconvergent systems a new level of flexibility.




In the last post, we talked about the fact that live migration is relevant for various operating systems, which, unlike containers, work continuously and contain a lot of useful data. However, today it is only one side of developments in this field. After all, applications with a long lifespan, such as database management systems, file storage, and so on, are no less sensitive to stopping.
')
Take for example a DBMS. Yes, you can kill it as a container and run it again. But how long it takes to repeat the download - only God knows. The process of recovering the cache from terabytes of various data may require a ton of additional resources. Needless to say that during its re-formation, the performance of all systems can be significantly reduced? But this process can take several hours. Another example is services serving long-running network connections. Alas, not all protocols provide the REST-style API today. In practice, there are many cases in which it is necessary to maintain a connection with the client for a long time. Restarting such an application is fraught with loss of access to the service.

Compel library

In the CRIU project, one very interesting possibility was used - the execution of a parasitic code by the process. This was necessary in order to prepare the container for migration. However, practice has shown that this possibility has much more applications, and we brought it to a separate library - Compel. Now anyone can write a program in C, compile it in a special way, and then take it and “load” it into the living process. She will work there, and then the victim process will return to her main job.



By the way, Compel also has our system of live application patching, which is the first “third-party” (alongside Criu itself) user of this library. It happens as follows: with the help of Compel, the launched program “patches” right during its operation. That is, you can install an updated version of the software on your machine and launch this updated version almost instantly.

There are two difficulties in this issue: first, you need to generate the “patch” itself, since the binary code is updated, that is, the executable instructions, and not the source code of the program. Secondly, you need to attach these changes so that the running program does not fall. It's akin to beating heart surgery. Until today, this solution was applied only to the OS kernel, since restarting the new kernel meant restarting the machine, and that was a long time. They didn’t do that to update applications, but if it’s possible for the kernel, why not for applications? And recently, we implemented a live patching technology, it has already been posted on Github.

static long process_syscall (struct process_ctx_s * ctx, int nr,

unsigned long arg1, unsigned long arg2,
unsigned long arg3, unsigned long arg4,
unsigned long arg5, unsigned long arg6)
{
int ret;
long sret = -ENOSYS;


ret = compel_syscall (ctx-> ctl, nr, & sret,
arg1, arg2, arg3, arg4, arg5, arg6);
if (ret <0) {
pr_err ("Failed to execute syscall% d in% d \ n", nr, ctx-> pid);
return ret;
}
if (sret <0) {
errno = -sret;
return -1;
}
return sret;
}


Sample code that makes a process make a system call

Technical details

Compel is a library for preparing, throwing into an arbitrary process, executing and unloading parasitic code. In order to do all this, Compel uses the debugger interface — the ptrace system call. Technically, it looks like this: Compel joins the process, stops it, then begins to rule its memory and registers.

However, it was not so easy to do all this. C stopping the process has its own subtleties: the process can be stopped without a debugger using the SIGSTOP signal. Therefore, for a long time, when working with the kernel, there was a serious problem: when the debugger was stopped (ptrace) of a process that was previously stopped by a signal (sigstop), the process would “wake up” as soon as the debugger was turned off. Of course, it was possible to send another STOP signal to the process before the debugger was turned off, and then it would be stopped anyway. But at the same time it was impossible to find out without dancing with a tambourine whether the process was stopped by a stop signal when connected or not. This situation is more or less acceptable for debugging, but not for a program that photographs the state of processes (i.e., for Criu or application patching). Especially for circumventing this point, an alternative method of stopping processes was developed, which did not lose information on whether it was stopped by a signal or not. This method is called PTRACE_SEIZE and today it is in all distribution kernels and is used including the strace utility.

int compel_interrupt_task (int pid)
{
int ret;

ret = ptrace (PTRACE_SEIZE, pid, NULL, 0);
if (ret) {
/ *
* ptrace API doesn't allow to distinguish
* attaching to zombie from other errors.
* All errors will be handled in compel_wait_task ().
* /
pr_warn ("Unable to interrupt task:% d (% s) \ n", pid, strerror (errno));
return ret;
}

/ *
* If we SEIZE-d
* and reading its stat from proc. Otherwise task
* may die _ while_ we're doing it
* inconsistent seize / state pair.
*
* If task dies after we
* do this interrupt, notice it via proc.
* /
ret = ptrace (PTRACE_INTERRUPT, pid, NULL, NULL);
if (ret <0) {
pr_warn ("SEIZE% d: can't interrupt task:% s", pid, strerror (errno));
if (ptrace (PTRACE_DETACH, pid, NULL, NULL))
pr_perror ("Unable to detach from% d", pid);
}

return ret;
}

Code for SEIZE

By the way, strace, trying to act in the old manner, if the operation SEIZE fails. But for CRIU - it is useless. If SEIZE does not work, the process state cannot be saved. We are sometimes asked whether it is possible to make CRIU work on those cores where there is no SEIZE. We say that theoretically it is possible, for this you will need to write support for SEIZE to Compel. However, this is not done deliberately, since then it will be impossible to guarantee the correct operation of Criu on the stopped processes.

There is one more nuance concerning signal processing. You can send a signal to the process stopped by the debugger, and the debugger itself will be woken up to handle it, which will decide what to do with the arriving signal. In the process of loading the parasitic code, Compel certainly runs into situations in which the “dissected” process currently receives signals from outside.

At first we tried to write code that could solve this situation, but it turned out to be too difficult to maintain, and with any changes there was a huge risk that signal processing would fail. So we decided to go the other way. Fortunately, Linux has the ability to block signals for a process, in which case debugging becomes much easier. However, the blocking interface is designed in such a way that the process can only block itself. You ask: we are loading a parasitic code into the process and we can block signals from it, what's the problem? But the problem is: while the parasite is loading, the signals need to be processed, and loading the parasite, as you understand, is quite complicated in itself, although the lack of the need to process signals after it does not greatly simplify the task.

To make life easier for themselves and, as it soon turned out, to the developers of the gdb debugger, a way was added to the kernel to block the signals of the process being debugged. This was done as another extension to the ptrace call. After that, the entire code for working with the parasite was greatly facilitated, but, alas, Compel (and Criu) lost the ability to work on kernels without this interface. However, unlike the operation SEIZE, to train Criu and Compel to work without the ability to block signals to an arbitrary process is possible, although it will require tremendous efforts.

static int arasite_run (pid_t pid, int cmd, unsigned long ip, void * stack,
user_regs_struct_t * regs, struct thread_ctx * octx)
{
k_rtsigset_t block;

ksigfillset (& block);
if (ptrace (PTRACE_SETSIGMASK, pid, sizeof (k_rtsigset_t), & block)) {
pr_perror ("Can't block signals for% d", pid);
goto err_sig;
}

parasite_setup_regs (ip, stack, regs);
if (ptrace_set_regs (pid, regs)) {
pr_perror ("Can't set registers for% d", pid);
goto err_regs;
}

if (ptrace (cmd, pid, NULL, NULL)) {
pr_perror ("Can't run parasite at% d", pid);
goto err_cont;
}

return 0;

err_cont:
if (ptrace_set_regs (pid, & octx-> regs))
pr_perror ("Can't restore regs for% d", pid);
err_regs:
if (ptrace (PTRACE_SETSIGMASK, pid, sizeof (k_rtsigset_t), & octx-> sigmask))
pr_perror (“Can't restore sigmask for% d”, pid);
err_sig:
return -1;
}

Signal blocking method for the process being debugged

Fortunately, today this problem has ceased to be acute. Both SEIZE and signal blocking became part of the Linux kernel functionality starting from 3.11 (and, of course, any newer versions), so the minimum system requirements for running Compel and Criu in particular are using the kernel version 3.11 or later.

Infect, use!


Currently, Compel is available at Github and can be used by anyone to run a parasitic code in any process. You can simply use the springboard in assembler, which helps to join the process and force it to do something - unload a part of the memory, replace it with data or update it. Today, there are many processes that would be good to fix without stopping, and Compel allows you to do it your way ... well, or you can use the ready-made utility for patching applications.