QNX RTOS: Inter-Task Interaction

Continuing the cycle of QNX real-time operating system notes. This time, I would like to talk about cross-tasking interaction in QNX Neutrino (we will consider QNX 6.5.0). In RTOS, there is a wide range of inter-tasking mechanisms — from QNX-specific messaging to familiar signals to developers of UNIX and POSIX and shared memory. And although most of the notes will be devoted to messaging, the features of using signals, POSIX messages and shared memory will also be described. And those who read to the end will get two buns for tea.

Understanding the messaging principle is essential for a QNX system programmer, since This mechanism plays a fundamental role in the RTOS. Many familiar and familiar to the developers of the functions of the operating system are only add-ons and implemented through messaging (for example, read() and write() ).

Qtx RTOS Interaction Forms

As mentioned above, messaging is a fundamental mechanism for inter-task interaction in QNX, on which several other mechanisms are based. All forms of inter-task interaction considered in this note are listed in the table below, indicating who is responsible for implementing this or that mechanism.
')
Table 1. Forms of inter-task interaction.

Mechanism	Scope of implementation
Message exchange	microkernel
Signals	microkernel
Shared memory	procnto process procnto
POSIX Message Queues	mqueue manager
Unnamed (pipe) and named (FIFO) program channels	pipe manager

In QNX RTOS 6.5.0, another form of inter-task interaction appeared - Persistent Publish / Subscribe (PPS). This is quite an interesting technology, which I will try to write about another time. Those interested can read the translation of the PPS section of the QNX Neutruno System Architecture.

Message exchange

This is a synchronous mezhzadachny interaction mechanism implemented in the QNX Neutrino micronucleus. When developing the RTOS, this form of inter-task interaction was not accidentally chosen as the main one. Firstly, the mechanism itself is quite simple. Secondly, the synchronous transmission and reception of messages facilitates debugging. Third, testing of high-level forms of interaction (for example, software channels) based on QNX Neutrino messaging and implemented in a monolithic core revealed approximately the same performance characteristics.

The messaging mechanism in QNX is also called the SRR mechanism, after the first letters of the three main functions used in messaging ¹ . MsgSend() is for sending a message, MsgReceive() is for receiving a message, and MsgReply() is for transmitting a response to the caller. First, consider each function separately to understand how they work, and then combine them in one example. All the arguments of the functions will not be intentionally cited so as not to be distracted and first understand the principle of operation.

MsgSend() is used to send a message from the client to the server and receive a response. The concepts of client and server are quite arbitrary here, because the same program can be a server for some tasks and, at the same time, a client of others. For example, a database server is a server for database clients ² . And at the same time, the database server will be the client for the file system manager. When the MsgSend() function is MsgSend() client is blocked in one of two states: SEND or REPLY . SEND status means that the client has sent a message, and the server has not yet received it. After the server receives the message, the client enters the REPLY state. When the server returns a response message, the client is unlocked.

MsgReceive() is used to receive messages from clients . The server calls MsgReceive() and is blocked in the RECEIVE state if none of the clients have yet sent him a message, i.e. did not call the MsgSend() function. After this happened (a message was sent to the server ). The server unlocks and continues its execution. The server usually needs to perform some actions to process the received message and prepare to receive a new one. If the server is working in several threads, then another thread can execute the message processing and response to the client . Most often, the thread receiving the message works in the "perpetual" cycle and after processing the received message, it calls MsgReceive() again.

MsgReply() used to send a response message to client ³ . When the MsgReply() function is MsgReply() lock does not occur, i.e. the server will continue to work. This is done because the client is already in a locked state ( REPLY ) and no additional synchronization is required.

What else needs to be learned in order to make up from those building blocks of knowledge that we have, a bridge to understanding the QNX messaging mechanism? Not so much. Be patient, now the picture will begin to take shape.

The QNX Neutrino microkernel does not care about the contents of the transmitted message. Messages do not have any format. The message makes sense only for the client and server . The microkernel only copies the message (i.e., just the data buffer) from the client 's address space to the server's address space (and vice versa when responding) and there is no intermediate buffer for storing messages. And that means there is no intermediate copy, because the microkernel copies data directly from the client ’s memory to the server’s memory (and vice versa when responding). As a result, the speed of the message passing mechanism increases.

To synchronize the transmission, reception, and response to a message, the microkernel blocks the threads involved in exchanging messages in one of three states: SEND , RECIEVE and REPLY . The client is blocked in the SEND state until the server accepts its message. The server is locked in the RECEIVE state if none of the clients sent a message to it. After receiving the message by the server , it is unlocked, and the client enters the locked state REPLY . After the server returns a response to the client , the latter is unlocked. That's all.

So, we figured out how the QNX messaging engine works. Fig. 1 simply illustrates the above.


Fig. 1. Messaging in QNX Neutrino.

How does the client find the server ?

If you understand how the messaging mechanism works in the QNX RTOS, you can move on. Probably, you should have one question, without deciding which, you will not be able to exchange messages in QNX. How does the client find the server ?

Messages are not transmitted directly between threads. Instead, channels and connections are used. The server creates a channel using the ChannelCreate() function. Now, finally, the server can call MsgReceive() and MsgReply() with a clear conscience. A piece of code below illustrates the operation of the server :

 chid = ChannelCreate( flags ); /*    chid  -1 */ for (;;) { rid = MsgReceive( chid, &msg, sizeof( msg ), NULL ); /*    rid  -1 */ switch ( msg.type ) { /*   */ } MsgReply( rid, EOK, NULL, 0 ); /*    ,    */ } 

In turn, the client creates a connection to the server channel using ConnectAttach() , and then calls MsgSend() . Client code is very simple:

 coid = ConnectAttach( nd, pid, chid, _NTO_SIDE_CHANNEL, 0 ); /*    coid  -1 */ /*   */ MsgSend( coid, smsg, sizeof( smsg ), rmsg, sizeof( rmsg ) ); /*    ,    */ /*   */ 

Now the last question remains. Where chid client find out the server parameters: nd , pid , chid ? These parameters are the server address or even the phone number with the city code and extension number. Half of the answer to this question is that the server itself knows all these parameters. But how can the server report them to the client ?

There are various ways to get this information from the server . You can use .pid files or global variables. But the right way for small applications is to use the name_attach() function in the server , and name_open() in the client . An even more correct way is to implement the server as a resource manager ⁴ , when it is responsible for the namespace element.

Composite messages

As already mentioned, one of the main advantages of the QNX Neutrino messaging engine is its high performance. This is achieved by the fact that there is no intermediate copying of data, i.e. The message is copied directly from the client ’s memory to the server’s memory. It often happens that these messages are in different places. A typical case is when the data is a raw buffer received from the hardware and a structure with information about the data (message header). The raw data itself may be located in a circular buffer. Is it really necessary in this case to prepare a separate buffer and copy the header and data from the ring buffer there? Wouldn't it be overkill? This unnecessary copying of data before sending a message can be avoided in the QNX RTOS by using composite messages.

To form composite messages in QNX Neutrino, you need to declare an array of type iov_t , with the number of elements equal (or more) to the number of messages, and initialize each element using the SETIOV() macro, i.e. specify the address and size of each buffer. Fig. 2 illustrates the principle of operation of composite messages.


Fig. 2. An example of a composite message.

To work with composite messages, the familiar functions of MsgReceive() and MsgReply() , but with the end of v, i.e. MsgReceivev() and MsgReplyv() . Since the function of sending a message MsgSend() also receives the result, it acquires a whole family: MsgSendv() , MsgSendsv() and MsgSendvs() . Now the microkernel will do all the extra work for us, and no additional copying and buffer with the whole message. I like that!

Impulses

Sometimes it is only required to inform another thread that something has happened, and no response is required. So it is not necessary and blocking on MsgSend() . In this case, the MsgSendPulse() function comes to the MsgSendPulse() . The pulse contains 8 bits of code and 32 bits of data. Very often, pulses are used in interrupt handlers. For pulses, queues are used, i.e. impulses will not be lost if the flow for some time did not accept them. But be prepared, sooner or later, to get an EAGAIN error if you send impulses to a stream that does not have time to read them.

Signals

The QNX Neutrino RTOS supports a signaling mechanism that should be familiar to UNIX developers. Both standard POSIX and real-time POSIX signals are supported. To work with both types of signals, the same microkernel code is used. As a result, the microkernel itself becomes more compact, and POSIX signals (at the request of an application) can be queued, just like their colleagues from the POSIX real-time signal group. Among other things, QNX Neutrino extends the POSIX standard and allows you to send signals to a specific stream. And this is sometimes very useful.

By the way, POSIX real-time signals contain 8 bits of code and 32 bits of data. Nothing like? Precisely, the same code that implements the signal mechanism is also used in the transmission of pulses. Convenient and reliable.

Signals use the familiar functions kill() , sigaction() , sigprocmask() , etc., as well as more interesting functions, for example, pthread_kill() and pthread_sigmask() .

Programming channels

Software channels should be familiar to UNIX users and developers. In QNX, all the familiar commands, functions and techniques also exist. For example, this is how an unnamed program pipe (pipe) is created on the command line:

 # ls -l | less 

To create a named pipe (FIFO), you must use the mkfifo command or the mkfifo() function.

There is only one feature. In order for QNX Neutrino to work with software channels, it is necessary to start the pipe manager.

POSIX Message Queues

Message queues are similar to named programmatic channels (FIFOs), but are more complex mechanisms because support message priorities. To work with POSIX message queues in QNX Neutrino, you must run the mqueue manager.

Note that message queues in the QNX RTOS can contain more than one slash '/', which means you can create directories. This extends the POSIX standard, which requires that the queue name begin with a slash and no longer contain this character. A rather convenient feature, since You can group the queues of one software package or one company in one directory.

Shared memory

Working with shared memory in QNX Neutrino is the same as in other UNIX systems. Since the shared memory mechanism is implemented in the procnto process procnto , which also contains the microkernel, you do not need to run anything else. You can read more in the documentation on the functions shm_open() and mmap() .

Separately, it should be noted that the shared memory itself is not suitable for inter-task interaction. Even if the server only writes to the shared memory, and the client only reads from it, it may happen that the client subtracts partially modified data. Such an error can be difficult to catch later, so it is better not to make it at all. To eliminate such a situation, it is necessary to apply one of the synchronization primitives, for example, mutexes or semaphores.

Not least of interest is the method of inter-task interaction, with the simultaneous use of the messaging mechanism and shared memory. This is especially useful if you plan to build a distributed system, because shared memory is not available over the network. This solution has a very high performance due to the use of shared memory, and also has the ability to synchronize and network transparency through the exchange of messages.

You should not try to build any interaction using shared memory, based on the fact that this is the fastest way. If synchronization primitives are used, then the speed can be comparable to the speed of the messaging mechanism. Significant gain will be only in the exchange of data of a very large amount.

Promised buns

Since I promised that there will be buns, they will. I did not forget, and I do not mind. The first thing is that when sending a message, you can use the same buffer for the message itself and the response to it. I would even say that they do it quite often. To make it more convenient, all structures describing various messages and the answers to them are grouped into one union. Normally, the message sent is not required by the client after receiving a response from the server . This way you can save on the buffer.

The second thing is that the POSIX file descriptor in QNX Neutrino is the same as the connection (for sending messages). And this means that functions that use file descriptors ( write() , read() , etc.) are just wrappers with minor overheads that convert their arguments into messages to the server . Serious enough optimization. So, if you want to develop system software for QNX, then learn to write a resource manager.

Bibliography

1. QNX Neutrino Real-Time Operating System 6.3. System architecture ISBN 5-94157-827-X
2. QNX Neutrino Real-Time Operating System 6.3. User's manual. ISBN 978-5-9775-0370-9
3. Rob Krten, “An Introduction to QNX Neutrino 2. A Guide for Real-Time Application Developers,” 2nd Edition. ISBN 978-5-9775-0681-6

---

¹ The prefix Msg in the name of the functions MsgSend() and others appeared only in QNX Neutrino, and in QNX4 these functions were called simply Send() , Receive() and Reply() . Hence the name SRR.

² In the example, the database server and clients do not use messaging functions directly, but we already know that in QNX, under each read() , write() , sendto() and others, the SRR mechanism is hidden.

³ If the server should return only an error code, for example, if the message is not supported, it is more convenient to use the MsgError() function.

^{4 A} description of resource managers is beyond the scope of this note. It is possible that I will write about this some other time.