Developing a free framework for developer needs is a specific topic. If at the same time the framework lives and develops for a long time, then the specifics are added. Today I will try to show it on the example of an attempt to extend the functionality of the "actor" framework for C ++ called SObjectizer .

The fact is that this framework is pretty old, changed dramatically several times. Even his current incarnation, SObjectizer-5, has undergone many changes, both serious and not so much. Plus, we are very sensitive to compatibility and making breaking changes to changes for us - this is too serious a step for it to be decided on just like that.
')
Right now we need to decide exactly how to add a new feature to the next version. In the process of finding a suitable solution emerged two options. Both look quite realizable. But very much different from each other. As for the complexity and complexity of the implementation, and in its "appearance." Those. what the developer will deal with in each of the options will look different. Probably even fundamentally different.

And here we, as framework developers, have to make a choice in favor of one or the other solution. Or it is necessary to recognize that none of them is satisfactory and, therefore, it is necessary to invent something else. Such decisions during the history of SObjectizer had to be made repeatedly. If it is interesting to someone to feel in the shoes of a developer of such a framework, then you are welcome under the cat.

Initial problem

So, briefly the essence of the original problem. In SObjectizer, from the very beginning of its existence, there was the following peculiarity: the timer message is not so easy to cancel. Under the timer will be understood further, first of all, the postponed message. Those. a message that should not be immediately sent to the recipient, but after some time. For example, we make send_delayed with a pause of 1s. This means that the message will actually be sent on a timer 1s after the send_delayed call.

Deferred message, in principle, can be canceled. If the message is still in the possession of the timer, the message after the cancellation will not go anywhere. It will be thrown by the timer and that's it. But if the timer has already sent the message and it is now in the order queue for the receiving agent, then canceling the timer will not work. It is not in the SObjectizer mechanism to remove a message from the request queue.

The problem is exacerbated by at least two factors.

First, 1: N mode is supported in SObjectizer, i.e. if the message was sent to the Multi-Consumer mbox, the message will not be in one queue, but in several queues for N recipients at once.

Secondly, in SObjectizer, the dispatcher mechanism is used and dispatchers can be very different, including those written by the user for their specific needs. Application queues are managed by dispatchers. And in the dispatcher interface there is no functionality for withdrawal of the application, which has already been transferred to the dispatcher. But even if such functionality in the interface would be incorporated, it is far from a fact that it could be implemented effectively in all cases. Not to mention that such functionality would increase the complexity of developing new controllers.

In general, objectively, if the timer has already sent a deferred message to the recipient (s), then you can’t force SObjectizer to deliver this instance of the message at the moment.

In fact, this problem is also relevant for periodic messages (that is, messages that the timer must send periodically at specified intervals). But in practice, the cancellation of periodic messages is needed much less frequently than the cancellation of a deferred message. At least in our practice it is.

What can be done right now?

So, this problem is not new and for a long time already there are recommendations on how to deal with it.

Unique id inside the deferred message

The easiest way is to keep a counter. The agent has a counter, when sending a deferred message in a message, the current value of the counter is sent. When a message is canceled, the agent counter is incremented. When a message is received, the current value of the counter in the agent is compared with the value from the message. If the values ​​do not match, the message is rejected:

class demo_agent : public so_5::agent_t { struct delayed_msg final { int id_; ... }; int expected_msg_id_{}; so_5::timer_id_t timer_; void on_some_event() { //   . //   send_periodic, ..    //  timer_id   . timer_ = so_5::send_periodic<delayed_msg>(*this, 25s, //     . 0s, //    . //      delayed_msg, //      id   . ++expected_msg_id_, ... //  . ); ... } void on_cancel_event() { //   ,        //   .   : timer_.reset(); //     . ++expected_msg_id_; //   id-. ... } void on_delayed_msg(mhood_t<delayed_msg> cmd) { //     id    //  . if(expected_msg_id_ == cmd->id_) { ... //  . } } }; 

The problem with this method is that the developer of the agent should be puzzled by the maintenance of these counters. And if, as a deferred message, we need to send someone else's message that someone else did, and in which there is no id_ field, then we find ourselves in a difficult situation.

Although, on the other hand, it is the most efficient way to exist.

Use unique mbox for deferred message

Another way that works well is to use a unique mailbox (mbox) for a deferred message. In this case, we create a new mbox for each deferred message, subscribe to it and send a deferred message to this mbox. When a message needs to be canceled, we simply delete the subscriptions for mbox.

 class demo_agent : public so_5::agent_t { struct delayed_msg final { ... //   id_   . }; so_5::mbox_t timer_mbox_; //   . so_5::timer_id_t timer_; void on_some_event() { //        mbox //     . timer_mbox_ = so_environment().create_mbox(); some_state.event(time_mbox_, ...); another_state.event(time_mbox_, ...); ... //    . timer_ = so_5::send_delayed<delayed_msg>( so_environment(), timer_mbox_, //     . 25s, 0s, ... //    delayed_msg. ); } void on_cancel_event() { //        mbox. timer_.reset(); so_drop_subscription_for_all_states(timer_mbox_); } void on_delayed_msg(mhood_t<delayed_msg> cmd) { //     ,   //    . ... } }; 

This method can already work with other people's messages, within which there is no unique identifier. But it also requires labor and attention from the developer.

For example, in the variant shown above, there is no protection against the fact that one deferred message has already been sent earlier. In an amicable way, before sending a new deferred message, you must always perform actions from on_cancel_event (), otherwise the agent will have unnecessary subscriptions.

Why was this problem not solved earlier?

It's all quite simple: in fact, it is not as serious a problem as it may seem. At least, in real life, it is not often necessary to encounter it. Usually, deferred and periodic messages are not canceled at all (which is why, by the way, the send_delayed function does not return timer_id). And when the need for cancellation arises, you can use one of the methods described above. Or even use some other. For example, create individual agents that will process the delayed message. These agents can be deregistered when a pending message needs to be canceled.

So, against the background of other tasks that we faced, the simplification of the guaranteed cancellation of the deferred message was not so priority as to spend its resources on solving this problem.

Why has the problem become urgent now?

It's just that simple. On the one hand, finally got around.

On the other hand, when SObjectizer starts using new people who have not had experience working with it, this feature with the cancellation of timers surprises them a lot. Not that pleasantly surprising. And if so, I would like to minimize the negative impressions of acquaintance with our tool.

In addition, we had our own tasks, we did not need to constantly cancel the deferred messages. And new users have their own tasks, maybe the opposite is true there.

New problem statement

Almost immediately, as soon as thinking about the possibility of a “guaranteed cancellation of the timer” began, the thought came to mind that the task could be expanded. You can try to solve the problem of recalling any of the previously sent messages, not necessarily deferred and periodic.

From time to time this opportunity is claimed. For example, imagine that we have several interacting agents of two types: entry_point (accepts requests from clients), and processor (processes requests):



The entry_point agents send requests to the agent, the processor processes them as far as possible and responds to the entry_point agents. But at times, the entry_point may detect that the processing of a previously sent request is no longer necessary. For example, the client sent a cancel command or the client “fell off” and it is no longer necessary to process his requests. Now, if the request messages are in the queue of the agent processor, then they cannot be revoked. And it would be useful.

Therefore, the current approach to solving the problem of "guaranteed cancellation of the timer" is performed exactly as the addition of support for "recall messages." We send any message in a special way, we get a certain handle with which we can then withdraw the message. And it is not so important whether a regular message or a delayed message is being recalled.

Attempting to come up with the implementation of "feedback messages"

So, you need to enter the concept of "revocable message" and support this concept in SObjectizer. And so, to stay within the framework of the 5.5 branch. The first version of this thread, 5.5.0, was released almost four years ago, in October 2014. Since then, there have been no major breaking changes in 5.5. Projects that have already been launched or immediately launched on SObjectize-5.5 can switch to new releases in branch 5.5 without any problems. This compatibility must be maintained at this time.

In general, everything is simple: you need to take and do.

What is clear how to do

After the first approach to the problem, two things became clear about the implementation of “recall messages”.

Atomic flag and its verification before processing the message

Firstly, it is obvious that within the framework of the current architecture of SObjectizer-5.5 (and perhaps more globally: within the framework of the principles of operation of SObjectizer-5 itself) it is impossible to withdraw messages from the dispatch queues of applications, where messages wait until the recipient agents process them. Attempting to do this will kill the whole idea of ​​disparate dispatchers, which even the user can do his own, for the specifics of his task (for example, like this ). In addition, in the case of sending a message in 1: N mode, where N is large, it will be expensive to store a list of pointers to the instance of the sent message in all queues.

This means that an atomic flag must be passed along with the message, which will need to be analyzed immediately after the message is retrieved from the request queue, but before the message is passed to the receiving agent for processing. Those. the message enters the queue and nowhere is it withdrawn from there. But when a message reaches the queue, its flag is checked. And if the flag says that the message has been withdrawn, the message is not processed.

Accordingly, the response of the message itself consists in setting a special value for the atomic flag within the message.

Object revocable_handle_t <M>

Secondly, while (?) It is obvious that not ordinary methods of sending messages, but the special object under the code name revocable_handle_t should be used to send a revocable message.

In order to send a revocable message, the user must create an instance of revocable_handle_t, then call the send method on that instance. And if the message needs to be recalled, this is done through the revoke method. Sort of:

 struct my_message {...}; ... so_5::revocable_handle_t<my_message> msg; //    . msg.send(target, //  . ... //    my_message. ); ... //   . msg.revoke(); 

There are no clear implementation details for revocable_handle_t, which is not surprising, since The mechanism of the work of recall messages has not yet been selected. But the principle of operation is that the revocable_handle_t retains a clever reference to the sent message and to the atomic flag for it. The revoke () method attempts to replace the flag value. If this succeeds, the message, after being retrieved from the queue, will not be processed anymore.

What it will not be friends

Unfortunately, there are a couple of things with which the feedback of the messages will not work out normally. Just because the withdrawn message continues to remain in those queues where it has already fallen.

message_limits

Such an important feature of SObjectizer, as message_limits , is designed to protect agents from overloading. Message_limits work based on counting messages in the queue. Put the message in the queue - increased the counter. We got out of the queue - reduced.

Since when a message is recalled, it continues to be in the queue, and the message_limits feedback is not affected by the message_limits. Therefore, it may happen that the queue is the maximum number of messages of type M, but they are all withdrawn. In fact, none of them will be processed. But to put a new message of type M in the queue does not work, because the limit will be exceeded.

The situation is not good. But how to get out of it? Unclear.

mchains with fixed queue size

In SObjectizer, a message can be sent not only to mbox, but also to mchain (this is our analogue of the CSP channel ). And mchains can have a fixed size of their queues. Attempting to put a new message for a fixed-size mchain in a full mchain should lead to some kind of reaction. For example, to wait for the release of space in the queue. Or to push the oldest message.

In the case of a message revocation, it will remain within the mchain queue. It turns out that the message is no longer needed, but it takes up space in the mchain queue. And prevents sending new messages to mchain.

The same bad situation, as in the case of message_limits. And again it is not clear how to fix it.

What is not clear how to do

So we got to choose between two (for now?) Implementation options for revocable messages. The first option is simple to implement and does not require reworking the offal of SObjectizer. The second option is much more complicated, but in it the recipient of the message does not even know that he is dealing with revocable messages. Briefly consider each of them.

Receiving feedback in the form of revocable_t <M>

The first solution, which looks, firstly, realizable and, secondly, quite practical, is the introduction of a special wrapper revocable_t <M>. When a user sends a revocation message of type M through revocable_handle_t <M>, then the message M is not sent, but the message M inside the special wrapper revocable_t <M>. And, accordingly, the user will receive and process not the message of type M, but the message revocable_t <M>. For example, in this way:

 class processor : public so_5::agent_t { public: struct request { ... }; // ,    . void so_define_agent() override { //   . so_subscribe_self().event( //     ,    //   . [this](mhood_t< revocable_t<request> > cmd) { // ,      . cmd->try_handle([this](mhood_t<request> msg) { ... }); }); ... } ... }; 

The revocable_t <M> :: try_handle () method checks the value of the atomic flag and, if the message is not withdrawn, calls the lambda function passed to it. If the message is withdrawn, try_handle () does nothing.

Pros and cons of this approach

The main advantage is that this campaign is easily implemented (at least for now it seems so). In fact, revocable_handle_t <M> and revocable_t <M> will be just a thin superstructure above SObjectizer.

Intervention in SObjectizer interiors may be required in order to make friends revocable_t and mutable_msg. The fact is that in SObjectizer there is the concept of immutable messages (they can be sent both in 1: 1 mode and in 1: N mode). And there is the concept of mutable messages that can only be sent in 1: 1 mode. In this case, the SObjectizer treats the mutable_msg <M> marker in a special way and performs the appropriate checks at run-time. In the case of revocable_t <mutable_msg <M >>, you will need to teach the SObjectizer to interpret this construction as mutable_msg <M>.

Another plus is that additional overhead costs (both for the metadata of the revocable message, and for checking the atomic flag) will only be in places where this cannot be done. In the same place where revocable messages are not used, there will be no additional overhead costs at all.

Well, the main minus ideological. In this approach, the use of revocable messages affects both the sender (using revocable_handle_t <M>) and the recipient (using revocable_t <M>). But just the recipient, and there is no need to know that he receives revocable messages. Moreover, as a recipient you may have a ready-made third-party agent, which is written without revocable_t <M>.

In addition, there remain ideological questions about, for example, the possibility of re-sending such messages. But, according to first estimates, these issues are solved.

Receiving feedback in the form of ordinary messages

The second approach is to see on the recipient side only a message of type M and not be aware of the existence of revocable_handle_t <M> and revocable_t <M>. Those. if the processor should receive the request, then it should see only the request, without any additional wrappers.

In fact, in this approach one cannot do without some wrappers, but they will be hidden inside the SObjectizer, and the user should not see them. After the application is removed from the queue, SObjectizer will determine that this is a specially wrapped revocation message, check the flag of the relevance of the message, expand the message if it is still relevant. After that, it will give the message to the agent for processing as if it were a regular message.

Pros and cons of this approach

The main advantage of this approach is obvious - the recipient of the message does not know what messages he is working with. This allows the message sender to quietly recall messages for any agents, even those that were written by other developers.

Another important plus is the ability to integrate with the message delivery tracing mechanism ( here the role of this mechanism is described in more detail ). Those. if msg_tracing is enabled and the sender revokes the message, then traces of this can be found in the msg_tracing log. Which is very convenient when debugging.

But the main disadvantage is the complexity of the implementation of this approach. When that will need to take into account several factors.

First, overhead. Different kind.

Let's say you can make a special flag inside a message that indicates whether the message is revocable or not. And then check this flag before processing each message. Roughly speaking, one more if is added to the message delivery mechanism, which will be processed when processing each (!) Message.

I’m sure that in real applications, the loss to this if will be hardly noticeable. But here drawdown in synthetic benchmarks will definitely appear. Moreover, the more abstract the benchmark, the less real work he does, the more he will squeeze. And this is bad from a marketing point of view, because There are a number of individuals who draw conclusions about the framework on indicators of synthetic benchmarks. Moreover, they do it specifically: without understanding what kind of benchmark it is, what it basically shows on what hardware it works, but comparing the totals with the performance of some specialized tool, in another scenario, on a different hardware, etc. ., etc.

In general, since we are doing a universal framework, which, so it turns out, is judged by abstract numbers in abstract benchmarks, you do not want to lose, say, 5% of the performance in the mechanism for delivering all messages due to the addition of a feature that will only need time from time to time and then not to all users.

Therefore, it is necessary to make sure that when sending a message to the recipient SObjectizer understand that when retrieving a message, you need to treat it in a special way. In principle, when a message is delivered to an agent, the SObjectizer stores along with the message a pointer to the function that will be used when processing the message. We need it now in order to handle asynchronous messages and synchronous requests in different ways. Actually, here is the application for a message addressed to the agent:

 struct execution_demand_t { //! Receiver of demand. agent_t * m_receiver; //! Optional message limit for that message. const message_limit::control_block_t * m_limit; //! ID of mbox. mbox_id_t m_mbox_id; //! Type of the message. std::type_index m_msg_type; //! Event incident. message_ref_t m_message_ref; //! Demand handler. demand_handler_pfn_t m_demand_handler; ... }; 

Where demand_handler_pfn_t is a regular function pointer:

 typedef void (*demand_handler_pfn_t)( current_thread_id_t, execution_demand_t & ); 

The same mechanism can also be used to specifically process the recalled message. Those. when mbox gives the agent a message, the agent knows whether an asynchronous message or a synchronous request is being sent to it. Similarly, an asynchronous revoked message can be given to the agent in a special way. And the agent will save along with the message a pointer to a function that knows how it should handle the revocable messages.

It seems to be all right, but there are two big "but" ... :(

Firstly, the existing interface mboxes (namely the class abstract_message_mbox_t ) does not have methods for sending feedback messages. So this interface needs to be expanded. And in such a way that other people's implementations of mboxes that are tied to abstract_message_box_t from SObjectizer-5.5 do not break (in particular, the mbox series is implemented in so_5_extra and I don’t want to break them).

Secondly, messages can be sent not only to mboxes, behind which agents are hidden, but also to mchains. Which are our analogs of CSP-shny channels . And there until now the application lay without any additional pointers to functions. Entering an additional pointer into each element of the request queue mchain, but it can, of course, but it looks like a rather expensive solution. In addition, the mchain implementations themselves have not yet provided for a situation in which the extracted message should be checked and, possibly, discarded.

If we try to summarize all the problems described above, the main problem with this approach is that it is not so easy to make its implementation so that it is cheap for cases when revocation messages are not used.

And what about the guaranteed cancellation of deferred messages?

I'm afraid the original problem was lost in the wilds of technical details. Suppose there are revocable messages, how will there be a cancellation of deferred / periodic messages?

Here, as they say, options are possible. For example, working with deferred / periodic messages can be part of the revocable_handle_t <M> functionality:

 revocable_handle_t<my_mesage> msg; msg.send_delayed(target, 15s, ...); ... msg.revoke(); 

Or you can make on top of revocable_handle_t <M> an additional helper class cancelable_timer_t <M>, which will provide the send_delayed / send_periodic methods.

White Spot: Synchronous Requests

SObjectizer-5 supports not only asynchronous interaction between entities in the program (via sending messages in mboxes and mchains), but also synchronous interaction through request_value / request_future. This synchronous interaction works not only for agents. Those.You can not only send a synchronous request to the agent through his mbox. In the case of mchains, you can also make synchronous requests, for example, to another working thread, on which you called receive () or select () for mchain.

So, it is still not clear whether to allow the use of synchronous requests in conjunction with revocable messages. On the one hand, there may be some meaning in this. And it may look, for example, like this:

 revocable_handle_t<my_request> msg; auto f = msg.request_future<my_reply>(target, ...); ... if(some_condition) msg.revoke(); ... f.get(); //      revoke(). 

On the other hand, there are still so many incomprehensible messages with revocable messages, so the issue of synchronous interaction has been postponed until better times.

Choose, but carefully. But choose

So, there is an understanding of the problem. There are two options to solve it. Which at the moment seem to be realizable. But they differ greatly in the level of convenience provided to the user, and even more they differ in the cost of implementation.

Between these two options to choose. Or come up with something else.

What is the difficulty of the choice?

The difficulty is that SObjectizer is a free framework. He does not bring us money directly. We do it, as they say, for our own. Therefore, purely from economic preferences, an easier and faster option to implement is more profitable.

But, on the other hand, not everything is measured by money, and in the long run, a quality-made tool, the features of which are normally linked to each other, is better than a patchwork of patches of some sort. Quality is assessed by both users and ourselves, when we subsequently accompany our development and add new features to it.

So the choice, in essence, is between short-term profit and long-term prospects. True, in the modern world, C ++ tools with long-term prospects are somehow vague. Which makes the choice even more difficult.

It is in such conditions that one has to choose. Caution. But choose.

Conclusion

In this article we tried to show a bit the process of designing and implementing new features in our framework. This process happens regularly with us. Earlier more often, because in 2014-2016, SObjectizer developed much more actively. Now the pace of release of new versions has decreased. What is objective, including because adding new functionality without breaking anything, with each new version becomes more difficult.

I hope it was interesting to see us backstage. Thanks for attention!