We continue to acquaint readers with the open C ++ framework called SObjectizer . Our framework simplifies the development of complex multi-threaded applications due to the fact that the C ++ programmer has access to higher-level tools borrowed from the Actor Model, CSP and Publish-Subscribe. At the same time, no matter how lofty it may sound, SObjectizer is one of the few open, living and developing action frameworks for C ++.

We have already devoted SObjectizer more than a dozen articles on Habré. But all the same, readers complain about the presence of "white spots" in understanding how SObjectizer works and how different types of entities that SObjectizer operates with are interrelated.

In this article we will try to look under the hood of the SObjectizer and we will try "on the fingers" and in the pictures to explain what it consists of and how, in general, it works.

SObjectizer Environment

Let's start with a thing like SObjectizer Environment (or SOEnv, if abbreviated). SOEnv is a container within which all entities related to a SObjectizer are created and work: agents, cooperatives, dispatchers, mailboxes, timers, etc. What can be illustrated with the following picture:
')


In fact, to start working with SObjectizer, you need to create and run an instance of SOEnv. For example, in this example, the programmer manually creates an instance of SOEnv as an object of type so_5 :: wrapped_env_t:

int main() { so_5::wrapped_env_t sobj{...}; ... // -   . return 0; } 

This instance will immediately start working and automatically terminate its work when the object so_5 :: wrapped_env_t is destroyed.

By itself, the essence of SOEnv as a separate concept was required in order to be able to run several independent SObjectizer instances within the same application.

 int main() { so_5::wrapped_env_t first_soenv{...}; so_5::wrapped_env_t second_soenv{...}; ... so_5::wrapped_env_t another_soenv{...}; ... // -   . return 0; } 

This makes it possible to get this picture in your application:

Funny fact. Our closest and much more raspiar competitor, the C ++ Actor Framework (also known as CAF), had not so long ago been able to run only one actor subsystem in an application. And we even came across a discussion in which CAF developers were asked why. But over time, the concept of actor_system and the ability to simultaneously run several actor_system in the application appeared in CAF.

What is the SObjectizer Environment responsible for?

In addition to the fact that SOEnv is a container that stores cooperations, dispatchers, etc., SOEnv also controls these entities.

For example, when starting SOEnv, the following should be started:

• a timer that will serve pending and periodic messages;
• the default dispatcher, on which all agents that were not explicitly associated with other dispatchers will work;
• user-created public dispatchers.

Accordingly, when stopping SOEnv, all these entities should be stopped.

Also, when a user wants to add his agents to SOEnv, the SOEnv must complete the registration procedure to cooperate with user agents. And when a user wants to withdraw his agents, SOEnv must deregister cooperation.

SOEnv has two main repositories that it owns and is responsible for. The first repository, which may disappear completely in the next major version of SObjectizer, is the repository of public dispatchers. Each public dispatcher must have its own unique string name, by which the dispatcher can be found and reused.

The second repository, which is the most important, is the cooperation repository. Each cooperation also must have its own unique string name, under which the cooperation is stored in the cooperation repository. Attempting to register a cooperation with a name already taken will result in an error.

It is possible that the presence of names in cooperatives is a rudiment inherited from SObjectizer-4. Currently, the names of cooperatives are considered as a rather ambiguous feature and, possibly, over time, the cooperation in the SObjectizer will become unnamed. But it is not exactly.

So, summarizing:

• SOEnv owns such entities as the timer, default and public dispatchers, cooperation;
• when starting, SOEnv starts the timer, default and public dispatchers;
• during operation, SOEnv is responsible for the registration and deregistration of cooperations;
• upon completion of the work, SOEnv deregisters all remaining live cooperatives, stops public and default dispatchers, and then stops the timer.

Environment Infrastructure

Inside SOEnv there is another interesting thing that makes SOEnv a more complex entity than it might seem. We are talking about SObjectizer Environment Infrastructure (or, abbreviated, env_infrastructure). To explain what it is and why, you need to talk about what interesting conditions we encountered as SObjectizer became used in completely different types of tasks.

When SObjectizer-5 appeared, SOEnv used multithreading to do its work. So, the timer was implemented by a separate timer thread. There was a separate working thread on which SOEnv completed the deregistration of cooperations and freed up all resources related to cooperations. And the default dispatcher was another working thread, on which applications linked to the default agent dispatchers were served.

Since SObjectizer is designed to simplify the implementation of complex multi-threaded applications, the use of multithreading within SObjectizer itself was considered (and is considered now) as a completely normal and acceptable solution.

However, as time went on, SObjectizer began to be used in projects that we ourselves hadn’t thought of before, and situations began to appear where multi-threaded SOEnv is too redundant and expensive.

For example, a small application that periodically wakes up, checks for the presence of some information, processes this information when it appears, adds the result somewhere and falls asleep again. All operations can be performed on a single workflow. In addition, the application itself should be lightweight and I would like to avoid the cost of creating additional workflows within SOEnv.

Another example: another small single-threaded application that actively works with the network through Asio. But at the same time, some part of the logic is easier to do not with Asio, but with SObjectizer agents. In this case, I would like to make both Asio and SObjectizer work in the same working context. Moreover, I would also like to avoid duplication of functionality: since Asio is used and Asio has its own timers, it makes no sense to run the same mechanism in SOEnv, even if SObjectizer uses timers from Asio to service deferred and periodic messages.

To make it possible to use SObjectizer, but also in such specific conditions, the notion of env_infrastructure appeared in SOEnv. At the C ++ level, env_infrastructure is an interface with some set of methods. When you start SOEnv, an object is created that implements this interface, after which SOEnv uses this object to do its work.



The SObjectizer includes several ready-made implementations of the env_infrastructure: ordinary multi-threaded; single-threaded, which is not thread-safe; single-threaded, which is thread-safe. Plus, in so_5_extra there are a couple of single-threaded Asio-based env_infrastructure - one thread-safe, and the second not. With a strong desire, the user can write his own env_infrastructure, although this is not an easy task, and also ungrateful, since we, the developers of SObjectizer, cannot guarantee that the env_infrastructure interface will remain unchanged. Too deep this thing integrates with SOEnv.

Agents, cooperations, dispatchers and disp_binder. And also event_queue

When working with a SObjectizer developer, one basically has to deal with the following entities:

• agents in which the business logic of the application (or part of the application) is implemented;
• dispatchers who determine how and where agents will work;
• messages and mailboxes through which agents exchange information between themselves and other parts of the application.

In this section we will talk about agents and dispatchers, and next we will go through mailboxes.

Dispatchers and event_queue

Let's start the conversation about agents with dispatchers, since having understood what dispatchers are for, it will be easier to understand the Venegrete of agents, agent cooperations and disp_binder.

The key point in implementing SObjectizer is that SObjectizer itself delivers messages to agents. The agent does not need to call a method of a receive in a loop, and then analyze the type of the message returned from the receive. Instead, the agent subscribes to messages of interest to him and when the necessary message arises, the SObjectizer itself calls the agent's handler method for the message.

However, the most important question in this scheme is this: where exactly does SObjectizer call the handler method? Those. on the context of which working thread does the agent process the messages addressed to it?

That's just the dispatcher - this is the very entity in SObjectizer, which is responsible for providing the working context for handling messages by agents. Roughly speaking, the dispatcher owns one or more worker threads, on which the agents' handler methods are called.

SObjectizer includes eight full-time dispatchers - ranging from the most primitive ones (for example, one_thread or thread_pool) to the advanced ones (like adv_thread_pool or prio_dedicated_threads :: one_per_prio). A developer can create as many dispatchers in his application as he needs.

For example, imagine that you need to make an application that will poll several devices connected to a computer, somehow process the received information, put it into a database and give this information to the outside world through some MQ broker. At the same time, interaction with devices will be synchronous, and data processing can be quite complex and multi-layered.

You can create one one_thread dispatcher for each device. Accordingly, all actions with the device will be performed on a separate thread and blocking this thread with a synchronous operation will not affect the rest of the application. Also, a separate one_thread-dispatcher can be allocated to work with the database. For the remaining tasks, you can create a single thread_pool dispatcher.



Thus, when a developer chooses a SObjectizer as a tool, one of the main tasks of the developer is to create the dispatchers necessary for the developer and associate the agents with the appropriate dispatchers.

event_queue

So, in SObjectizer, the agent does not need to independently determine if there is any message waiting to be processed. Instead, the dispatcher itself, to which the agent is attached, calls the agent handler methods for the incoming agent messages.

But here the question arises: how does the dispatcher know that some message is addressed to the agent?

The question is not idle, because in the "classical" Model of Actors, each actor has its own queue of messages addressed to the actor. In the first versions of SObjectizer-5, we went down the same path: each agent had its own message queue. When the message was sent to the agent, the message was saved in this queue, and then the dispatcher, to which the agent was attached, was asked to process this message. It turned out that sending a message to an agent required replenishment of two queues: the message queue of the agent itself and the queue of dispatcher requests.

This scheme had its positive aspects, but all of them were leveled by a huge disadvantage - its inefficiency. Therefore, soon in SObjectizer-5, we abandoned our own message queues with agents. Now the queue in which messages addressed to the agent are placed belongs not to the agent, but to the dispatcher.

The logic is simple, if the dispatcher determines where and when the agent will process its messages, then let the dispatcher own the message queue of the agent. So now in SObjectizer the following picture takes place:



The connecting element between the agent and the dispatcher is the event_queue, an object with a specific interface that saves the agent’s message to the appropriate dispatch request list.

The event_queue object is owned by the dispatcher. It is the dispatcher that determines how exactly event_queue is implemented, how many event_queue objects it will have, whether event_queue will be unique for each agent, or several agents will work with a common event_queue object, etc.

The agent initially has no connection with the dispatcher, this link appears at the moment of binding the agent to the dispatcher. After the agent is attached to the dispatcher, the agent gets a reference to the event_queue and when the agent is sent a message addressed to the agent, this message is sent to the event_queue and the event_queue is already responsible for ensuring that the request for processing the message is placed in the required dispatcher queue.

In this case, there are several moments in the life of an agent when the agent does not have contact with the dispatcher, i.e. The agent does not have a link to its event_queue. The first point is the gap between the creation of the agent and its association with the dispatcher at the time of registration. The second point is the period of deregistration, when the agent is already untied from the dispatcher, but not yet destroyed. If at these moments the message is addressed to the agent, then in the process of its delivery it is found that the agent does not have an event_queue and the message in this case is simply discarded.

Agents, cooperations and disp_binder

Starting agents in SObjectizer-e occurs in four stages.

At the first stage, the programmer creates an empty collaboration (more details below).

At the second stage, the programmer creates an instance of his agent. The agent in SObjectizer-e is implemented by the usual C ++ class and the creation of the agent is performed like the usual creation of an instance of this class.

At the third stage, the programmer must add his agent to the cooperation. Cooperation is another unique thing that, as far as we know, is only in SObjectizer. A co-operation is a group of agents that must appear in SOEnv and disappear from SOEnv simultaneously and transactionally. That is, if there are three agents in cooperation, then all three must successfully start their work in SOEnv, or none of them should do this. In the same way, either all three agents are simultaneously withdrawn from SOEnv, or all three agents continue to work together.

The need for cooperation arose almost immediately at the beginning of work on the SObjectizer, when it became clear that in most cases agents would be created in an application, not one by one, but interrelated groups. And so that the developer did not have to invent the control schemes for the start of the group and the implementation of the rollback, when two of the three agents he needed started successfully, and the third did not, and co-operations were invented.

So, in the third step, the programmer fills his cooperation with agents. After the cooperation is full, follow the fourth step - registration of cooperation. In code, it might look like this:

 so_5::environment_t & env = ...; // SOEnv     . //  №1:  . auto coop = env.create_coop("demo"); //  №2:  ,      . auto a = std::make_unique<my_agent>(... /*   my_agent*/); //  №3:    . coop->add_agent(std::move(a)); ... //  №4:  . env.register_coop(std::move(coop)); 

When registering a cooperation, the developer transfers the created and completed cooperation to SOEnv. SOEnv performs a number of actions: checks the uniqueness of the name of the cooperation, requests the dispatchers the resources necessary for the cooperation agents, calls the agents so_define_agent () method, binds the agents to the dispatchers, sends each agent a special message so that the agent is called so_evt_start () . Naturally, with the rollback of previously performed actions, if any operation from this list failed.

And when the cooperation is registered, then the agents are already inside the SObjectizer (more precisely, within the specific SOEnv) and can work fully.

One of the most important parts of the registration of cooperation is the binding of agents to dispatchers. After binding, the agent has a real link to event_queue, which makes it possible to deliver messages to the agent.

After successful registration of cooperation, we will have some such picture:

disp_binder

Above, we have already mentioned “binding agents to dispatchers” several times, but have never explained how this binding is performed. How does the SObjectizer understand in general, to which particular dispatcher each agent should be tied to?

And here comes the special objects called disp_binder. They serve precisely to bind the agent to the dispatcher when registering cooperation with the agent. And also in order to untie the agent from the dispatcher during the deregistration of cooperation.

In SObjectizer, an interface is defined that all disp_binder should support. The specific implementations of disp_binder depend on the specific type of dispatcher. And each dispatcher implements its own disp_binder.

To bind the agent to the dispatcher, the developer must create disp_binder and indicate this disp_binder when the agent is added to the collaboration. In essence, the code for filling the cooperation should look something like this:

 auto & disp = ...; //   ,     . env.introduce_coop("demo", [&](so_5::coop_t & coop) { //    ,  disp_binder  . coop.make_agent_with_binder<my_agent>(disp->binder(), ... /*    my_agent */); ... }); 

The important point: it is the cooperation that owns the disp_binder, and only the cooperation knows which agent uses the disp_binder. Therefore, the real picture of the registered cooperation will look like this:

Mailboxes

Another key element of SObjectizer, which makes sense to consider at least superficially, is mailboxes (or mboxes, in the terminology of SObjectizer).

The presence of mailboxes also distinguishes SObjectizer from other actor frameworks that implement the “classical” Model of Actors. In the “classical” Model of Actors, messages are addressed to a specific actor. Therefore, the sender of the message must know the link to the recipient actor.

In SObjectizer, the legs grow not only (and not so much) from the Model of Actors, but also from the Publish-Subscribe mechanism. Therefore, we have the operation of sending a message in 1: N mode is initially built into SObjectizer. And therefore in SObjectizer messages are not sent directly to agents, but in mboxes. For the mbox-ohm can be a single agent recipient. Or several (or several hundred thousand recipients). Or none at all.

Since the messages are not sent directly to the recipient agent, but to the mailbox, we needed to enter another concept that is not in the “classical” Model of Actors, but which is the cornerstone of Publish-Subscribe: a subscription to messages from mbox.In SObjectizer, if an agent wants to receive messages from the mbox, he must subscribe to the message. No subscription - no messages reach the agent. There is a subscription - they make it.

Regular types of mboxes

There are two types of mboxes in SObjectizer. The first type is multi-producer / multi-consumer (MPMC). This type of mboxes is used to implement M: N mode interaction. The second type is multi-producer / single-consumer (MPSC). This type of mbox-s appeared later and it is intended for effective interaction in the M: 1 mode.

Initially, in SObjectizer-5 there were only MPMC-mboxes, since the M: N delivery mechanism is enough to solve any problems. And those that require interaction in M: N mode, and those that require interaction in M: 1 mode (in this case, a separate mbox is created, which is owned by a single recipient). But in M: 1 mode, MPMC-mboxes have too high overheads compared to MPSC-mboxes, therefore, MPSC-mboxes were added to SObjectizer for reducing the overheads for M: 1 interaction cases.

. MPSC-mbox- SObjectizer , . , , . MPSC-mbox- .

Multi-Producer/Multi-Consumer mbox-

MPMC-mbox is responsible for delivering the message to all agents that have subscribed to the message. There will be many such agents, whether there will be such an agent in the singular or there will be no such agents at all - these are just the details of the work. Therefore, MPMC-mbox stores a list of subscribers for each type of message. And the general scheme of MPMC-mbox can be represented as follows:



Here Msg1, Msg2, ..., MsgN are the types of messages that agents subscribe to.

Multi-Producer / Single-Consumer mboxes

MPSC-mbox is much simpler than MPMC-mbox, so it works more efficiently. The MPSC-mbox-e only stores the link to the agent with which this MPSC-mbox is associated:

The mechanism of message delivery to the agent "on the fingers"

If you tell quite briefly how messages in SObjectizer are delivered to the recipient, the following picture appears: The



message is sent to the mbox. Mbox selects the recipient (in the case of MPMC-mbox, it is all subscribers to this type of message, in the case of MPSC-mbox, it is the sole owner of mbox, a) and returns the message to the recipient.

The recipient looks to see if he has a current link to event_queue. If so, the message is sent to the event_queue. If there is no reference to event_queue, the message is ignored.

If the message was sent to event_queue, then event_queue stores the message in the corresponding dispatcher queue. What this queue will be depends on the type of dispatcher.

When the dispatcher reaches this message when scrapping his queues, he will call the agent on his working context (roughly speaking the context of some of his working threads). The agent will find a handler method for this message and call it (we emphasize again, the call will occur on the context provided by the dispatcher).

That, in fact, is all that can be said about the principle of operation of the message delivery mechanism in SObjectizer in general terms. Although everything is somewhat more complicated there, but today we will not look at the very details.

Conclusion

In this article we tried to make a clear, albeit superficial, overview of the main mechanisms and features of the SObjectizer. We hope that this article will help someone to better understand how SObjectizer works. And, perhaps, it is better to understand what SObjectizer can be useful for.

But if you do not understand something or want to learn more about something, then ask questions in the comments. We like it when people ask us questions and we answer them with pleasure. At the same time, many thanks to all those who ask questions - you force us to improve and develop both SObjectizer itself and the documentation for it.

Also, taking this opportunity, we want to offer everyone who is not yet familiar with SObjectizer to get acquainted with our framework. It is written in C ++ 11 (minimum requirements for gcc-4.8 or VC ++ 12.0), it works under Windows, Linux, FreeBSD, macOS and, with the help of CrystaxNDK, on ​​Android-e. It is distributed under the BSD-3-CLAUSE license (i.e. free of charge). You can take with github-a or SourceForge . Currently available documentation is here . Plus, a large number of examples are included in SObjectizer, and yes, they are all up to date :)

Look, suddenly like it. And if something does not like it, then let us know, we will try to fix it. Feedback is very important to us now, so if you haven’t found something necessary for yourself in SObjectizer, then tell us about it. Perhaps we can add this in the next versions of SO-5.