Understanding interprocess communication on Linux

Inter-process communication (IPC) is a set of methods for exchanging data between process threads. Processes can be run on the same computer or on different, connected by a network. There are several types of IPC: “signal”, “socket”, “semaphore”, “file”, “message” ...

In this article I want to consider only 3 types of IPC:

1. named pipe
2. shared memory
3. semaphore

Retreat: this article is educational and is intended for people who are just entering the system programming path. Its main idea is to get acquainted with various ways of interaction between processes on a POSIX-compatible OS.

Named Channel

To send messages, you can use the mechanisms of sockets, channels, D-bus and other technologies. You can read about sockets at every corner, and write a separate article about D-bus. Therefore, I decided to dwell on obscure technologies that meet POSIX standards and give working examples.

Consider messaging on named pipes. Schematically, the transfer looks like this:

To create named pipes, we will use the function mkfifo () :

#include <sys/stat.h> int mkfifo(const char *pathname, mode_t mode); 

The function creates a special FIFO file named pathname , and the mode parameter sets the file permissions.
')

Note: mode is used in conjunction with the current umask value as follows: ( mode & ~ umask) . The result of this operation will be the new umask value for the file we are creating. For this reason, we use 0777 ( S_IRWXO | S_IRWXG | S_IRWXU ) to not overwrite any bits of the current mask.

Once a file is created, any process can open this file for reading or writing as well as opening a regular file. However, for correct use of the file, it is necessary to open it simultaneously with two processes / threads, one to receive data (read the file), the other to transfer (write to the file).

Upon successful creation of the FIFO file, mkfifo () returns 0 (zero). In case of any errors, the function returns -1 and sets the error code to the variable errno .

Typical errors that may occur during channel creation:

• EACCES - no rights to run (execute) in one of the directories in the pathname path
• EEXIST - pathname file already exists, even if the file is a symbolic link
• ENOENT - there is no directory mentioned in pathname , or it is a broken link
• ENOSPC - no place to create a new file.
• ENOTDIR - one of the directories mentioned in the pathname is not really one
• EROFS - an attempt to create a FIFO file on a read-only file system

Reading and writing to the created file is performed using the read () and write () functions.

Example

mkfifo.c

 #include <sys/stat.h> #include <fcntl.h> #include <string.h> #include <stdio.h> #define NAMEDPIPE_NAME "/tmp/my_named_pipe" #define BUFSIZE 50 int main (int argc, char ** argv) { int fd, len; char buf[BUFSIZE]; if ( mkfifo(NAMEDPIPE_NAME, 0777) ) { perror("mkfifo"); return 1; } printf("%s is created\n", NAMEDPIPE_NAME); if ( (fd = open(NAMEDPIPE_NAME, O_RDONLY)) <= 0 ) { perror("open"); return 1; } printf("%s is opened\n", NAMEDPIPE_NAME); do { memset(buf, '\0', BUFSIZE); if ( (len = read(fd, buf, BUFSIZE-1)) <= 0 ) { perror("read"); close(fd); remove(NAMEDPIPE_NAME); return 0; } printf("Incomming message (%d): %s\n", len, buf); } while ( 1 ); } 

[ download ]

We open the file as read only ( O_RDONLY ). And they could use the O_NONBLOCK modifier, designed specifically for FIFO files, so as not to wait for the file to be opened on the other side. But in the above code, this method is inconvenient.

Compile the program, then run it:

 $ gcc -o mkfifo mkfifo.c $ ./mkfifo 

In the next terminal window we execute:

 $ echo 'Hello, my named pipe!' > /tmp/my_named_pipe 

As a result, we will see the following output from the program:

 $ ./mkfifo /tmp/my_named_pipe is created /tmp/my_named_pipe is opened Incomming message (22): Hello, my named pipe! read: Success 

Shared memory

The next type of interprocess communication is shared memory . We schematically depict it as a certain named area in memory, which is accessed simultaneously by two processes:

To allocate shared memory, we will use the POSIX shm_open () function:

 #include <sys/mman.h> int shm_open(const char *name, int oflag, mode_t mode); 

The function returns the file descriptor that is associated with the memory object. This descriptor can later be used by other functions (for example, mmap () or mprotect () ).

The integrity of the memory object is preserved, including all data associated with it, until the object is disconnected / deleted ( shm_unlink () ). This means that any process can access our memory object (if it knows its name) until we explicitly call shm_unlink () in one of the processes.

The variable oflag is the bitwise "OR" of the following flags:

• O_RDONLY - open with read access only
• O_RDWR - open with read and write permissions
• O_CREAT - if the object already exists, then no effect from the flag. Otherwise, an object is created and access rights for it are set in accordance with mode.
• O_EXCL — setting this flag in conjunction with O_CREATE will result in an error returning with the shm_open function if the shared memory segment already exists.

How the value of the mode parameter is specified is described in detail in the previous paragraph “message passing”.

After creating a shared memory object, we set the size of the shared memory by calling ftruncate () . At the entrance of the function, the file descriptor of our object and the size we need.

Example

The following code demonstrates the creation, modification, and deletion of shared memory. It also shows how after creating shared memory, the program exits, but the next time we start we can access it, until shm_unlink () is executed.

shm_open.c

 #include <unistd.h> #include <sys/types.h> #include <sys/mman.h> #include <fcntl.h> #include <stdio.h> #include <string.h> #define SHARED_MEMORY_OBJECT_NAME "my_shared_memory" #define SHARED_MEMORY_OBJECT_SIZE 50 #define SHM_CREATE 1 #define SHM_PRINT 3 #define SHM_CLOSE 4 void usage(const char * s) { printf("Usage: %s <create|write|read|unlink> ['text']\n", s); } int main (int argc, char ** argv) { int shm, len, cmd, mode = 0; char *addr; if ( argc < 2 ) { usage(argv[0]); return 1; } if ( (!strcmp(argv[1], "create") || !strcmp(argv[1], "write")) && (argc == 3) ) { len = strlen(argv[2]); len = (len<=SHARED_MEMORY_OBJECT_SIZE)?len:SHARED_MEMORY_OBJECT_SIZE; mode = O_CREAT; cmd = SHM_CREATE; } else if ( ! strcmp(argv[1], "print" ) ) { cmd = SHM_PRINT; } else if ( ! strcmp(argv[1], "unlink" ) ) { cmd = SHM_CLOSE; } else { usage(argv[0]); return 1; } if ( (shm = shm_open(SHARED_MEMORY_OBJECT_NAME, mode|O_RDWR, 0777)) == -1 ) { perror("shm_open"); return 1; } if ( cmd == SHM_CREATE ) { if ( ftruncate(shm, SHARED_MEMORY_OBJECT_SIZE+1) == -1 ) { perror("ftruncate"); return 1; } } addr = mmap(0, SHARED_MEMORY_OBJECT_SIZE+1, PROT_WRITE|PROT_READ, MAP_SHARED, shm, 0); if ( addr == (char*)-1 ) { perror("mmap"); return 1; } switch ( cmd ) { case SHM_CREATE: memcpy(addr, argv[2], len); addr[len] = '\0'; printf("Shared memory filled in. You may run '%s print' to see value.\n", argv[0]); break; case SHM_PRINT: printf("Got from shared memory: %s\n", addr); break; } munmap(addr, SHARED_MEMORY_OBJECT_SIZE); close(shm); if ( cmd == SHM_CLOSE ) { shm_unlink(SHARED_MEMORY_OBJECT_NAME); } return 0; } 

[ download ]

After creating the memory object, we set the size we need to shared memory by calling ftruncate () . Then we accessed shared memory using mmap () . (Generally speaking, even using the mmap () call itself, you can create shared memory. But the difference with the shm_open () call is that the memory will remain allocated until the computer is deleted or restarted.)

Compile the code this time with the -lrt option:

 $ gcc -o shm_open -lrt shm_open.c 

Look what happened:

 $ ./shm_open create 'Hello, my shared memory!' Shared memory filled in. You may run './shm_open print' to see value. $ ./shm_open print Got from shared memory: Hello, my shared memory! $ ./shm_open create 'Hello!' Shared memory filled in. You may run './shm_open print' to see value. $ ./shm_open print Got from shared memory: Hello! $ ./shm_open close $ ./shm_open print shm_open: No such file or directory 

The “create” argument in our program is used both to create shared memory and to change its contents.

Knowing the name of the memory object, we can change the contents of the shared memory. But once we call shm_unlink () , the memory is no longer available to us and shm_open () without the O_CREATE parameter returns the error "No such file or directory".

Semaphore

A semaphore is the most commonly used method for synchronizing threads and controlling simultaneous access by multiple threads / processes to shared memory (for example, a global variable). The interaction between processes in the case of semaphores is that the processes work with the same data set and adjust their behavior depending on this data.

There are two types of semaphores:

1. counting semaphore, defining the limit of resources for processes accessing them
2. binary semaphore (binary semaphore), having two states “0” or “1” (more often: “busy” or “not busy”)

Consider both types of semaphores.

Semaphore with counter

The meaning of a semaphore with a counter is to give access to some resource only to a certain number of processes. The rest will wait in line when the resource is free.

So, to implement semaphores, we will use the POSIX sem_open () function:

 #include <semaphore.h> sem_t *sem_open(const char *name, int oflag, mode_t mode, unsigned int value); 

In the function to create a semaphore, we pass the name of the semaphore, built according to certain rules and control flags. Thus we will have a named semaphore.
The name of the semaphore is constructed as follows: at the beginning is the symbol "/" (slash), followed by Latin characters. The “slash” symbol should no longer be used. The length of the semaphore name can be up to 251 characters.

If we need to create a semaphore, then the O_CREATE control flag is passed . To start using an existing semaphore, the oflag is zero. If, together with the O_CREATE flag, to pass the O_EXCL flag, the sem_open () function will return an error if the semaphore with the specified name already exists.

The mode parameter sets permissions in the same way as explained in previous chapters. And the value variable initializes the initial value of the semaphore. Both mode and value are ignored when a semaphore with the specified name already exists, and sem_open () is called with the O_CREATE flag.

To quickly open an existing semaphore, we use the construction:

 #include <semaphore.h> sem_t *sem_open(const char *name, int oflag); 

where only the semaphore name and control flag are indicated.

An example of a semaphore with a counter

Consider an example of using a semaphore to synchronize processes. In our example, one process increases the value of the semaphore and waits for the second to reset it in order to continue further execution.

sem_open.c

 #include <fcntl.h> #include <sys/stat.h> #include <semaphore.h> #include <stdio.h> #define SEMAPHORE_NAME "/my_named_semaphore" int main(int argc, char ** argv) { sem_t *sem; if ( argc == 2 ) { printf("Dropping semaphore...\n"); if ( (sem = sem_open(SEMAPHORE_NAME, 0)) == SEM_FAILED ) { perror("sem_open"); return 1; } sem_post(sem); perror("sem_post"); printf("Semaphore dropped.\n"); return 0; } if ( (sem = sem_open(SEMAPHORE_NAME, O_CREAT, 0777, 0)) == SEM_FAILED ) { perror("sem_open"); return 1; } printf("Semaphore is taken.\nWaiting for it to be dropped.\n"); if (sem_wait(sem) < 0 ) perror("sem_wait"); if ( sem_close(sem) < 0 ) perror("sem_close"); return 0; } 

[ download ]

In one console, run:

 $ ./sem_open Semaphore is taken. Waiting for it to be dropped. <--       sem_wait: Success sem_close: Success 

In the next console, run:

 $ ./sem_open 1 Dropping semaphore... sem_post: Success Semaphore dropped. 

Binary semaphore

Instead of a binary semaphore, for which the sem_open function is also used, I’ll consider a semaphore that is often called a “mutex” (mutex).

A mutex is essentially the same as a binary semaphore (i.e. a two-state semaphore: busy and not busy). But the term “mutex” is often used to describe a scheme that prevents two processes from simultaneously using common data / variables. While the term “binary semaphore” is more commonly used to describe a construct that restricts access to a single resource. That is, a binary semaphore is used where one process “occupies” the semaphore, and the other “releases” it. While the mutex is released by the same process / thread that occupied it.

Without writing a mutex, for example, a database that many clients can access.

To use a mutex, you must call the pthread_mutex_init () function:

 #include <pthread.h> int pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *mutexattr); 

The function initializes the mutex (variable mutex ) with the mutexattr attribute. If mutexattr is NULL , then the mutex is initialized with the default value. In case of successful execution of the function (return code 0), the mutex is considered initialized and “free”.

Common mistakes that may occur:

• EAGAIN - insufficient resources (except memory) for mutex initialization
• ENOMEM - not enough memory
• EPERM - no rights to perform an operation
• EBUSY - an attempt to initialize a mutex that has already been initialized, but not undefined
• EINVAL - mutexattr value is not valid.

To occupy or free a mutex, use the functions:

 int pthread_mutex_lock(pthread_mutex_t *mutex); int pthread_mutex_trylock(pthread_mutex_t *mutex); int pthread_mutex_unlock(pthread_mutex_t *mutex); 

The pthread_mutex_lock () function, if the mutex is not yet occupied, then it occupies it, becomes its owner, and immediately exits. If the mutex is busy, it blocks further process execution and waits for the mutex to be released.
The pthread_mutex_trylock () function is identical in the behavior of the pthread_mutex_lock () function, with one exception — it does not block the process if mutex is busy, but returns an EBUSY code.
The pthread_mutex_unlock () function frees a busy mutex.

Return codes for pthread_mutex_lock () :

• EINVAL - mutex incorrectly initialized
• EDEADLK - the mutex is already taken by the current process.

Return codes for pthread_mutex_trylock () :

• EBUSY - mutex is already taken
• EINVAL - mutex incorrectly initialized

Return codes for pthread_mutex_unlock () :

• EINVAL - mutex incorrectly initialized
• EPERM - the calling process is not the owner of the mutex

Mutex example

mutex.c

 #include <stdio.h> #include <pthread.h> #include <unistd.h> #include <errno.h> static int counter; // shared resource static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; void incr_counter(void *p) { do { usleep(10); // Let's have a time slice between mutex locks pthread_mutex_lock(&mutex); counter++; printf("%d\n", counter); sleep(1); pthread_mutex_unlock(&mutex); } while ( 1 ); } void reset_counter(void *p) { char buf[10]; int num = 0; int rc; pthread_mutex_lock(&mutex); // block mutex just to show message printf("Enter the number and press 'Enter' to initialize the counter with new value anytime.\n"); sleep(3); pthread_mutex_unlock(&mutex); // unblock blocked mutex so another thread may work do { if ( gets(buf) != buf ) return; // NO fool-protection ! Risk of overflow ! num = atoi(buf); if ( (rc = pthread_mutex_trylock(&mutex)) == EBUSY ) { printf("Mutex is already locked by another process.\nLet's lock mutex using pthread_mutex_lock().\n"); pthread_mutex_lock(&mutex); } else if ( rc == 0 ) { printf("WOW! You are on time! Congratulation!\n"); } else { printf("Error: %d\n", rc); return; } counter = num; printf("New value for counter is %d\n", counter); pthread_mutex_unlock(&mutex); } while ( 1 ); } int main(int argc, char ** argv) { pthread_t thread_1; pthread_t thread_2; counter = 0; pthread_create(&thread_1, NULL, (void *)&incr_counter, NULL); pthread_create(&thread_2, NULL, (void *)&reset_counter, NULL); pthread_join(thread_2, NULL); return 0; } 

[ download ]

This example demonstrates the sharing of two threads to a common variable. One thread (the first thread) in the automatic mode constantly increases the counter variable by one, while taking up this variable for a full second. This first thread gives the second access to the count variable for only 10 milliseconds, then again takes it for a second. In the second stream, it is proposed to introduce a new value for the variable from the terminal.

If we didn’t use the mutex technology, then what value would be in the global variable, with the simultaneous access of two threads, we do not know. Also during startup, the difference between pthread_mutex_lock () and pthread_mutex_trylock () becomes apparent.

Compile the code with the optional -lpthread parameter:

 $ gcc -o mutex -lpthread mutex.c 

We start and change the value of a variable by simply entering a new value in the terminal window:

 $ ./mutex Enter the number and press 'Enter' to initialize the counter with new value anytime. 1 2 3 30 <Enter> <---    Mutex is already locked by another process. Let's lock mutex using pthread_mutex_lock(). New value for counter is 30 31 32 33 1 <Enter> <---    Mutex is already locked by another process. Let's lock mutex using pthread_mutex_lock(). New value for counter is 1 2 3 

Instead of conclusion

In the following articles I want to look at d-bus and RPC technologies. If there is interest, let me know.
Thank.

UPD: Updated the 3rd chapter on semaphores. Added a subchapter about mutex.