Listen up
After all, if the stars are lit - it means - is it necessary for someone?

V.V. Mayakovsky, 1914

I do programming for embedded systems, and decided to write this article in order to better understand the problem of using the fork () and vfork () system calls. The second of them is often advised not to use, but it is clear that he appeared for a reason.

Let's see when and why it is better to use this or that challenge.
')
As a bonus, the description of vfork () / fork () implementations in our project will be given. First of all, my interest is connected with the use of these calls in embedded systems, and the main feature of these implementations is the absence of virtual memory. Perhaps habrovchane well versed in system programming and embedded systems, will give advice and share experiences.

Who cares, I ask under the cat.

Let's start with the definition, that is, the POSIX standard, in which these functions are defined:

fork () creates an exact copy of the process with the exception of a few variables. If successful, the function returns zero to the child process and the number of the child process to the parent (the processes then begin to “live their life”).

vfork () is defined as fork () with the following restriction: the behavior of the function is not defined if the process created with its help performs at least one of the following actions:

• Will return from the function in which vfork () was called;
• Call any function except _exit () or exec * () ;
• Changes any data except the variable that stores the value returned by the vfork () function.

In order to understand why there is a system call at all with such strong limitations, you need to figure out what an exact copy of the process is.

One of the first links in the search engine for this topic in Russian is the description of the cloning parameters of processes in Linux. It follows that some parameters can be made common for the parent and child processes:

• Address space (CLONE_VM);
• File system information (CLONE_FS);
• Open file table (CLONE_FILES);
• Table of signal handlers (CLONE_SIGHAND);
• Parent process (CLONE_PARENT).

In POSIX for vfork (), it is not allowed to change variables, and this suggests that it is a matter of cloning the address space. This link confirms the assumption:

Unlike fork () , vfork () does not create a copy of the parent process, but creates an address space that is shared with the parent process until the _exit function or one of the exec functions is called.
The parent process at this time stops its execution. All the restrictions on use follow from this — the child process cannot change any global variables or even common variables that are shared with the parent process.

In other words, if this statement is true, after calling vfork () both processes will see the same data.
Let's do an experiment. If this is true, then the changes made to the data of the child process must be visible in the parent process, and vice versa.


If you build and run this program, we get the output:
fork (): common variable hasn't been changed.

When replacing fork () with vfork () , the output will change:
vfork (): common variable has changed.

Many use this property when transferring data between processes, although the behavior of such programs is not defined by POSIX. This is likely to create problems that make it advised not to use vfork () .

Indeed, one thing is when a developer consciously changes the value of some variable, and quite another when he forgets that the child process cannot, for example, return from the function in which vfork () was called (because it will destroy the stack structure of the parent process !) And even acting consciously, as usual, you use undocumented features at your own risk.

But a couple of less obvious problems:

• The book “Secure Programming for Linux and Unix HOWTO” says that even if a child does not really change any data in a high-level language code, it may not be so in computer code (for example, due to the appearance of hidden temporary variables).
• This blog analyzes the following question: what if vfork () is called in a multithreaded application? Consider the vfork () implementation in Linux: the manual says that the parent process stops when it is called, but in fact it only happens with the current thread (which, of course, is easier to implement). This means that the child process continues to run in parallel with other threads that may, for example, change the rights of the parent process. And here everything will become very bad: we will get two processes with different rights in the same address space, which opens a security hole.

Now consider the functions of the exec * family . Only they (apart from _exit () ) can be called in the process obtained with vfork () . They create a new address space, and then load the code and data from the specified file into it. In this case, the old address space, in fact, is destroyed.
Therefore, if the process is created using fork () and then calls exec * () , creating (copying) the address space when calling fork () was redundant, and this is quite a laborious operation, and it may take the most time to call fork. () In Wikipedia, for example, this moment is given the most attention, and, unlike the standard, it is directly stated :

Space for the child. Copy of the parent process.

Of course, on most modern systems with virtual memory, no copying takes place; all pages in the memory of the parent process are simply flagged with copy-on-write . However, at the same time you need to go over the entire hierarchy of tables, and this takes time.

It turns out that the vfork () call should execute faster than fork () , which is also mentioned in the LinuxMan page .

We will conduct another experiment and make sure that this is true. Let's slightly change the previous example: add a loop to create 1000 processes, remove the common variable and display it on the screen.


Run through the time command.

Output when using fork ()	Output when using vfork ()
real 0m0.135s user 0m0.000s sys 0m0.052s	real 0m0.028s user 0m0.000s sys 0m0.016s

The result, to put it mildly, is impressive. From launch to launch, the data will differ slightly, but still vfork () will be 4 to 5 times faster.

The conclusions are as follows:
fork () is a more “heavy” call, and if vfork () can be called, it is better to use it.
vfork () is a less secure call, and it’s easier to shoot yourself in the foot, and, accordingly, it should be applied intelligently.
fork () / vfork () should be used where you need to create separate resources for the process (inodes, user, working folder), otherwise it is worth working with pthread *, which work even faster.
It is better to use fork () when you really need to create a separate address space. However, it is very difficult to implement on small processor platforms without hardware support for virtual memory.

Before turning to the second part of the article, I note that in POSIX there is a function posix_spawn () . This function, in fact, contains vfork () and exec () , and therefore avoids the problems associated with vfork () , in the absence of re-creating the address space as in fork () .

Now let's move on to our fork () / vfork () implementation without MMU support.

Vfork implementation

Implementing vfork () in our system, we assumed that the vfork () call should occur like this: the parent goes into standby mode, and the first of vfork () returns the child process, waking up the parent when calling the _exit () or exec * () function. This means that a descendant can be executed on the parent stack, but with its own resources of other types: inodes, signal table, and so on.

The storage of various types of resources in our project is a task ( struct task ). It is this structure that describes all the resources of the process, including the available memory, inodes, and the list of threads that belong to this process. A task always has a main thread - the one that is created when it is initialized. The flow in our system is called the object of planning, more about this - in the article of my colleague . Since the stack is controlled by a thread, not a task, we can offer two options for implementation:

• Change the stack in the newly created thread to the parent's stack;
• “Replace” a task with a new one for the same execution thread

One way or another, the task will have to be created, or rather, to inherit it from the parent one: a clone of the signal table, environment variables, and so on will be made. Address space, however, will not be inherited.

Return from vfork () will be performed twice: for the parent and child processes. It means that the registers of the stack frame from which vfork () was called should be saved somewhere. You cannot do this on the stack, since the child process can wipe these values ​​at runtime. However, the vfork () signature does not imply the presence of some kind of buffer, so first the registers are stored on the stack, and only then - somewhere in the parent task. Saving registers on the stack could be done using a system call, but we decided to do without it and did it ourselves. Naturally, the vfork () function is written in assembler.

Thus, the registers are first stored on the stack, and then the C- shny function vfork_body () is called. As an argument, it is passed a pointer to a structure with a set of registers.

The vfork_body () code is architecturally independent. He is responsible for creating the task and saving the registers needed to exit.

A few explanations to the code.
First, a multithreading check occurs (problems associated with it when using vfork () , discussed above). Then a new task is created, and if it succeeds, the registers are saved in it to return from vfork () .
After that, the vfork_child_start () function is called , which, as the name implies, “starts” the child process. The quotes here are not random, as in fact the task can be launched later, it all depends on the specific implementation, of which there are two in our project. Before proceeding to their description, consider the functions _exit () and exec * () .
When they are called, the parent thread must be unblocked. We will say that at this very moment a full-fledged launch of the child process occurs as a separate entity in the system.

Other functions of the exec * family are expressed through the execv () call.


As can be seen from the above code, in order to unlock the parent process, the vfork_child_done () function is used with the handler as one of the parameters. To implement a particular algorithm, the work must be implemented:

• vfork_child_start () - the function called at the beginning of the cloning process should block the parent process;
• vfork_child_done () is a function that is called upon the final start of the child process, the parent process is unlocked;
• task_exit_callback () - a handler for completing a child process;
• task_exec_callback () - a handler for the full launch of the child process.

First implementation

The idea of ​​the first implementation is to use the same control flow besides the same stack. In fact, in this case, you only need to “replace” the task for the current thread with the child one until the child task finally starts when you call vfork_child_done () .


The following happens: the current execution thread (that is, the parent thread) is bound to the child process by the thread_set_task () function — for this, it suffices to change the corresponding pointer in the structure of the current thread. This means that when accessing resources associated with a task, the thread will refer to the child's task, and not to the parent, as before. For example, when a thread tries to find out which task a thread belongs to (the task_self () function), it will receive a child task.

After this, the child task is marked as created as a result of vfork , this flag will be needed in order for the vfork_child_done () function to be executed as needed (more details - a little later).
Then the registers saved by the vfork () call are restored. Recall that according to POSIX, the vfork () call should return a value of zero to the child process, which is done by calling ptregs_retcode_jmp (ptregs, 0) .

As already mentioned, when the child process calls the _exit () or execv () function, the vfork_chlid_done () function must unblock the parent stream. In addition, you need to prepare a child task for executing the required handler.




When calling vfork_child_done (), you must consider the case of using exec () / _exit () without vfork () - then you just need to exit the current function, because there is no need to unlock the parent, and you can immediately proceed to launch the child. If the process was created using vfork () , the following is done: first, the is_vforking flag is removed from the child task using task_vfork_end () , then, finally, the main thread of the child task starts. The entry point is the run function, which should be one of the handlers described earlier ( task_exec_callback , task_exit_callback ) - they are necessary when implementing vfork () . After that, the thread’s belonging to the task changes: instead of the child, the parent is specified again. Finally, it returns to the parent task from the vfork () call with the child process ID as the return value. As mentioned above, this is done by calling ptregs_retcode_jmp () .

The second implementation of vfork

The idea of ​​the second implementation is to use the parent stack with a new thread that was created with the new task. This is obtained automatically if the registers previously saved in the parent stream are restored in the child stream. In this case, you can use this synchronization between threads, as described in the already mentioned article . This is certainly more beautiful, but also more difficult to implement, because when the parent thread is waiting, its descendant will be executed on the same stack. This means that for the waiting time, you need to switch to some intermediate stack, where you can safely wait for the call of a child of _exit () or exec * () .


Explanation of the code:
First, space is allocated for the stack, after that the pid (process ID) of the child is saved, since it will be required by the parent to return from vfork () .
Calling setjmp () will return to the stack location where vfork () was called. As already mentioned, the wait must be performed on some intermediate stack, and the switch is performed using the CONTEXT_JMP_NEW_STACK () macro, which changes the current stack and transfers control to the vfork_waiting () function. It will activate the child and block the ancestor before calling vfork_child_done () .


As can be seen from the code, first of all the table of signals of the child process is saved. In fact, the SIGCHLD signal will be redefined, which is sent when the status of the child process changes. In this case, it is used to unlock the parent.


Saving and restoring the signal table is done using the POSIX call sigaction () .



After replacing the signal handler, the task is marked as being in the standby mode, in which it will remain until the present launch of the child task when calling _exit () / exec * () . The vfork_child_task () function is used as the entry point to the task, which restores previously saved registers and returns from vfork () .


When calling _exit () and exec * (), SIGCHLD will be sent, and the handler of this signal will uncheck waiting for the child to run. After that, the old SIGCHLD signal handler will be restored, and the control will be returned to the vfork_child_start () function using longjmp () . It must be remembered that the stack frame of this function will be damaged after the execution of the child process, therefore local variables will contain not what is needed. After releasing the previously allocated stack, the vfork () function returns the number of the child task.

Vfork health check

To check the correct behavior of vfork (), we wrote a set of tests covering several situations.

Two of them check for a valid return from vfork () when calling _exit () and execv () as a child process.


Another test checks the use of the same stack by the parent and child processes.


However, I would like to check the correctness of the work on some real, and third-party, program, and for this, a fairly well-known dropbear package was chosen . When configured, it checks for fork () , and if it does not find it, it can use vfork () . Immediately I would like to say that this was done in order to support ucLinux , and not in order to improve performance.

The OS was configured accordingly (for dropbear to use vfork () ), and using ssh to successfully connect to both implementations.



PS Also in our project we managed to implement fork () itself without using the MMU, at the moment an article is being compiled about it.