This article is a continuation of a reflection article published a month ago, " Is it easy to add new features to the old framework? Flour of choice on the example of the development of SObjectizer ". In that article, the task we wanted to solve in the next version of SObjectizer was described, two approaches to its solution were considered, and the advantages and disadvantages of each approach were listed.

Time passed, one of the approaches was brought to life and the new versions of SObjectizer , as well as the accompanying project so_5_extra , already what is called “ breathing deeply ”. You can literally take and try.
')
Today we will talk about what was done, why it was done, what it led to. If it is interesting to someone to follow the development of one of the few living, cross-platform and open actor frameworks for C ++, you are welcome under cat.

How did it all start?

It all started with an attempt to solve the problem of guaranteed cancellation of timers. The essence of the problem is that when a deferred or periodic message is sent, the programmer can cancel the delivery of the message. For example:

auto timer_id = so_5::send_periodic<my_message>(my_agent, 10s, 10s, ...); ... // - . // ,    my_message    . timer_id.release(); //      my_message. 

After calling timer_id.release (), the timer will no longer send new instances of the my_message message. But those copies that have already been sent and hit the queue of recipients will not go anywhere. Over time, they will be extracted from these queues and will be transferred to recipient agents for processing.

This problem is a consequence of the basic principles of operation of SObjectizer-5 and does not have a simple solution due to the fact that SObjectizer cannot remove messages from the queues. It cannot, because in SObjectizer the queues belong to the dispatchers, the dispatchers are different, their queues are also organized in different ways. Including there are dispatchers who are not part of the SObjectizer and SObjectizer, in principle, can not know how these dispatchers work.

In general, there is such a feature among native timers SObjectizer. Not that she too spoiled the life of developers. But some extra attention needs to be shown. Especially for beginners who are just familiar with the framework.

And then, finally, the hands went so far as to propose a solution for this problem.

What solution was chosen?

In the previous article , two possible options were considered. The first option did not require modification of the message delivery mechanism in SObjectizer, but instead required the programmer to explicitly change the type of message sent / received.

The second option required a modification of the SObjectizer message delivery mechanism. It was this path that was chosen because it allowed the fact that the message was sent in a specific way to be hidden from the recipient of the message.

What has changed in SObjectizer?

New concept: envelope with message inside

The first component of the implemented solution is the addition of a concept such as an envelope to SObjectizer. An envelope is a special message, within which lies the actual message (payload). SObjectizer delivers an envelope with a message to the recipient in almost the usual way. The principal difference in the processing of the envelope is found only at the very last stage of delivery:

• upon delivery of a regular message, the receiving agent simply searches for a handler for this type of message and, if such a handler is found, the found handler is called and the delivered message is given as a parameter;
• and upon delivery of the envelope with the message, after the handler is found, an attempt is first made to get the message from the envelope. And only if the envelope has given the message stored in it, only then is the handler called.

There are two key points that have a major impact on what and how envelopes with messages can be used.

The first key point is that a message is requested from an envelope only when a handler is found for the message from the receiver. Those. only when the message is indeed delivered to the recipient and the recipient will process this message right here and now.

The second key point here is that the envelope may not give the message in it. Ie, for example, the envelope can check the current time and decide that all delivery times have been missed and, therefore, the message has ceased to be relevant and cannot be processed. Therefore, the envelope will not give the message out. Accordingly, SObjectizer will simply ignore this envelope and will not take any additional actions.

What is an envelope?

An envelope is an implementation of the envelope_t interface, which is defined as follows:

 class SO_5_TYPE envelope_t : public message_t { public: ... // -. //   ,       //    . virtual void handler_found_hook( handler_invoker_t & invoker ) noexcept = 0; //   ,      //     . virtual void transformation_hook( handler_invoker_t & invoker ) noexcept = 0; private : kind_t so5_message_kind() const noexcept override { return kind_t::enveloped_msg; } }; 

Those. An envelope is essentially the same message as everyone else. But with a special sign, which is returned by the so5_message_kind () method.

The programmer can develop his envelopes by inheriting from envelope_t (or, which is more convenient, from so_5 :: extra :: enveloped_msg :: just_envelope_t ) and overriding the hook methods handler_found_hook () and transformation_hook ().

Inside the hook methods, the envelope developer decides whether he wants to send the message inside the envelope for processing / transformation or not. If so, the developer should call the invoke () method and the invoker object. If he does not want, then he does not call, in this case the envelope and its contents will be ignored.

How do envelopes solve the problem of canceling timers?

The solution that is now implemented in so_5_extra as the so_5 :: extra :: revocable_timer namespace is very simple: with a special sending of a deferred or periodic message, a special envelope is created, inside which is not only the message itself, but the atomic revoked flag. If this flag is cleared, the message is considered relevant. If set, the message is considered withdrawn.

When a hook method is called on an envelope, the envelope checks the value of the revoked flag. If the flag is set, the envelope does not give the message out. Thus, the message processing is not performed even if the timer has already managed to place the message in the recipient's queue.

Interface extension abstract_message_box_t

Adding the envelope_t interface is only one part of the envelope implementation in SObjectizer. The second part is taking into account the fact of the existence of envelopes in the message delivery mechanism inside the SObjectizer.

Here, unfortunately, it was not without making changes visible to the user. In particular, in the class abstract_message_box_t, which defines the interface of all mailboxes in SObjectizer, it was necessary to add another virtual method:

 virtual void do_deliver_enveloped_msg( const std::type_index & msg_type, const message_ref_t & message, unsigned int overlimit_reaction_deep ); 

This method is responsible for delivering the message envelope to the recipient with a message like msg_type inside. Such delivery may vary in implementation details depending on what the mbox is.

When adding do_deliver_enveloped_msg () to abstract_message_box_t, we had a choice: to make it a pure virtual method or to suggest some default implementation.

If we made do_deliver_enveloped_msg () a pure virtual method, then we would break the compatibility between the SObjectizer versions in branch 5.5. After all, then those users who wrote their own mbox implementations would have to modify their own mboxes when switching to SObjectizer-5.5.23, otherwise they would not be able to compile with the new version of SObjectizer.

We did not want this, so we did not do do_deliver_enveloped_msg () as a pure virtual method in v.5.5.23. It has a default implementation that simply throws an exception. Thus, custom user mboxes can continue to work normally with regular messages, but will automatically refuse to accept envelopes. We found this behavior more acceptable. Moreover, at the initial stage it is unlikely that envelopes with messages will be used widely, and it is unlikely that custom implementations of SObjectizer mboxes are often found in the “wild” nature;)

In addition, there is a far from zero chance that in subsequent major versions of SObjectizer, where we will not look at compatibility with branch 5.5, the interface abstract_message_box_t will undergo major changes. But we are already running far ahead ...

How to send envelopes with messages

SObjectizer 5.5.23 itself does not provide simple means of sending envelopes. It is assumed that a specific type of envelope and appropriate tools are developed for a specific task for conveniently sending envelopes of a particular type. An example of this can be seen in so_5 :: extra :: revocable_timer , where you need not only to send an envelope, but also to give the user a special timer_id.

For simpler situations, you can use the tools from so_5 :: extra :: enveloped_msg . For example, this is how sending a message with a specified limit on its delivery time looks like:

 // make     . so_5::extra::enveloped_msg::make<my_message>(... /*    */) // envelope         . //  5s        . .envelope<so_5::extra::enveloped_msg::time_limited_delivery_t>(5s) //        . .send_to(destination); 

To make it really fun: envelopes in envelopes

Envelopes are designed to carry inside some messages. But which ones?

Any.

And this brings us to an interesting question: is it possible to put an envelope inside another envelope?

Yes you can. How much you want. The nesting depth is limited only by the common sense of the developer and the stack depth for the recursive call handler_found_hook / transformation_hook.

At the same time, SObjectizer goes to meet the developers of their own envelopes: the envelope should not think about what is inside it - a specific message or another envelope. When a hook method is called on an envelope and the envelope decides that it can return its contents, the envelope simply invokes handler_invoker_t and transmits a reference to its contents to invoke (). And already invoke () inside itself will understand what it deals with. And if this is another envelope, then invoke () itself will call the necessary hook method for this envelope.

Using the above toolkit from so_5 :: extra :: enveloped_msg, the user can make several nested envelopes like this:

 so_5::extra::enveloped_msg::make<my_message>(...) // ,        my_message. .envelope<inner_envelope_type>(...) // ,      inner_envelope_type. .envelope<outer_envelope_type>(...) .send_to(destination); 

Some examples of using envelopes

Now, after we have walked through the insides of SObjectizer-5.5.23, it’s time to move on to the more useful for users, application part. Below are a few examples that are either based on what is already implemented in so_5_extra, or use tools from so_5_extra.

Recall timers

Since the whole kitchen with envelopes was started in order to solve the problem of guaranteed recall of timer messages, let's see what happened in the end. We will use the example from so_5_extra-1.2.0, which uses the tools from the new so_5 :: extra :: revocable_timer namespace:


What do we have here?

We have an agent who first initiates several timer messages, and then blocks his working thread for a while. During this time, the timer manages to queue an agent for several requests as a result of timers that have been triggered: several instances of periodic, one instance of first_delayed and second_delayed.

Accordingly, when the agent unlocks its thread, it should receive the first periodic and first_delayed. When processing the first_delayed agent, it cancels the delivery of periodic-second and second_delayed. Therefore, these signals to the agent should not reach regardless of whether they are already in the agent's queue or not (and they are).

We look at the result of the example:

 hang the agent...done periodic received first_delayed received last_delayed received 

Yes, that is right. Received the first periodic and first_delayed. Then there is neither periodic, nor second_delayed.

But if in the example we replace the “timers” from so_5 :: extra :: revocable_timer with regular timers from SObjectizer, then the result will be different: the agent will be reached by those instances of the periodic and second_delayed signals that have already reached the agent in the queue.

Messages with restrictions on delivery time

Another useful thing at times that will become available in so_5_extra-1.2.0 is the delivery of a message with a time limit. For example, the request_handler agent sends a verify_signature message to the crypto_master agent. At the same time, request_handler wants verify_signature to be delivered within 5 seconds. If this does not happen, then there will be no sense in handling verity_signature, the request_handler agent will already stop its work.

And the crypto_master agent is a friend who loves to be a bottleneck: sometimes it starts to slow down. At such moments, messages such as the above verify_signature accumulate in the queue, which can wait until the crypto_master becomes relieved.

Suppose that request_handler sent the message verify_signature to the crypto_master agent, but then crypto_master poplohe oh he "stuck" for 10 seconds. The request_handler agent has already fallen off, i.e. already sent everyone a denial of service and completed their work. But the message verify_signature in the crypto_master queue remains! So, when the crypto_master "otlipnet", then he will take this message and will process this message. Although it is no longer needed.

With the help of the new envelope so_5 :: extra :: enveloped_msg :: time_limited_delivery_t, we can solve this problem: the request_handler agent sends verify_signature nested in the envelope time_limited_delivery_t with a delivery time limit:

 so_5::extra::enveloped_msg::make<verify_signature>(...) .envelope<so_5::extra::enveloped_msg::time_limited_delivery_t>(5s) .send_to(crypto_master_mbox); 

Now, if the crypto_master "sticks" and does not have time to get to verify_signature in 5 seconds, then the envelope simply will not give this message for processing. And crypto_master will not do work that no one needs.

Message delivery reports to the recipient

And finally, an example of a curious thing, which is not implemented regularly either in SObjectizer, or in so_5_extra, but which can be done independently.

Sometimes you want to get something like a “delivery report” from a SObjectizer message to a recipient. Indeed, it is one thing when the message reached the recipient, but the recipient, for whatever his reasons, did not react to it. Another thing is when the message did not reach the recipient at all. For example, it was blocked by an overload protection agent mechanism . In the first case, the message to which we did not wait for an answer, you can not resend. But in the second case, it may make sense to re-send the message after some time.

Now we will look at how, using envelopes, you can implement the simplest “delivery reports” mechanism.

So, first we will do the necessary preparatory actions:

 #include <so_5_extra/enveloped_msg/just_envelope.hpp> #include <so_5_extra/enveloped_msg/send_functions.hpp> #include <so_5/all.hpp> using namespace std::chrono_literals; namespace envelope_ns = so_5::extra::enveloped_msg; using request_id_t = int; 

Now we can define the messages that will be used in the example. The first message is a request to perform any actions we need. And the second message is a confirmation that the first message reached the recipient:

 struct request_t final { request_id_t m_id; std::string m_data; }; struct delivery_receipt_t final { //   request_t::m_id   request_t. request_id_t m_id; }; 

Next, we can define a processor_t agent that will process request_t messages. But will process with imitation of "sticking". Those. it handles request_t, after which it changes its state from st_normal to st_busy. In the st_busy state, it does nothing and ignores all messages that arrive to it.

This means that if the processor_t agent sends three request_t messages in a row, it will process the first one, and the other two will be thrown, because when processing the first message, the agent will go to st_busy and ignore what will come to him while he is in st_busy.

In st_busy, the processor_t agent will spend 2 seconds, after which it will return to st_normal again and will be ready to process new messages.

Here’s how the agent processor_t looks like:

 class processor_t final : public so_5::agent_t { //   .     //   . state_t st_normal{this, "normal"}; //  " ".   . state_t st_busy{this, "busy"}; public: processor_t(context_t ctx) : so_5::agent_t{std::move(ctx)} { this >>= st_normal; st_normal.event(&processor_t::on_request); //     ,    . //  2   ,    st_normal. st_busy.time_limit(2s, st_normal); } private: void on_request(mhood_t<request_t> cmd) { std::cout << "processor: on_request(" << cmd->m_id << ", " << cmd->m_data << ")" << std::endl; this >>= st_busy; } }; 

Now we can define the agent requests_generator_t, which has a packet of requests that need to be delivered to the processor_t. The agent request_generator_t sends the entire pack every 3 seconds, and then waits for delivery confirmation in the form of delivery_receipt_t.

When the delivery_recept_t arrives, the requests_generator_t agent throws the delivered request out of the packet. If the bundle is completely empty, then the work of the example is completed. If there is still something left, the remaining bundle will be sent again when the next reshipment time comes.

So here is the request_generator_t agent code. It is quite voluminous, but primitive. You can only pay attention to the inside of the send_requests () method, which sends request_t messages nested in a special envelope.


Now we have messages and there are agents who must communicate with these messages. Only a small amount remains - to somehow force the delivery_receipt_t messages to arrive when the request_t is delivered to the processor_t.

This is done with the help of such an envelope:

 class custom_envelope_t final : public envelope_ns::just_envelope_t { //     . const so_5::mbox_t m_to; // ID  . const request_id_t m_id; public: custom_envelope_t(so_5::message_ref_t payload, so_5::mbox_t to, request_id_t id) : envelope_ns::just_envelope_t{std::move(payload)} , m_to{std::move(to)} , m_id{id} {} void handler_found_hook(handler_invoker_t & invoker) noexcept override { //    ,     . //     . so_5::send<delivery_receipt_t>(m_to, m_id); //      . envelope_ns::just_envelope_t::handler_found_hook(invoker); } }; 

In general, there is nothing complicated. We inherit from so_5 :: extra :: enveloped_msg :: just_envelope_t. This is an auxiliary envelope type that stores the message attached to it and provides a basic implementation of the
handler_found_hook () and transformation_hook () hooks . Therefore, we can only save the necessary attributes inside custom_envelope_t and send delivery_receipt_t inside the handler_found_hook () hook.

That's all. If we run this example, we get the following:

 sending request: (0, First) sending request: (1, Second) sending request: (2, Third) sending request: (3, Four) processor: on_request(0, First) request delivered: 0 time to resend requests, pending requests: 3 sending request: (1, Second) sending request: (2, Third) sending request: (3, Four) processor: on_request(1, Second) request delivered: 1 time to resend requests, pending requests: 2 sending request: (2, Third) sending request: (3, Four) processor: on_request(2, Third) request delivered: 2 time to resend requests, pending requests: 1 sending request: (3, Four) processor: on_request(3, Four) request delivered: 3 

In addition, it must be said that in practice such a simple custom_envelope_t is hardly suitable for generating delivery reports. But if someone is interested in this topic, then it can be discussed in the comments, and not to increase the volume of the article.

What else could be done with envelopes?

Great question!To which we ourselves do not yet have an exhaustive answer. Probably, the possibilities are limited except by the fantasy of users. Well, if something is missing for the realization of fantasies in SObjectizer, then you can tell us about it. We always listen. And, importantly, sometimes we even do :)

Integration of agents with mchain

If we speak a little more seriously, then there is another feature that I would like to have at times and which was even planned for so_5_extra-1.2.0. But which, most likely, will not get into release 1.2.0 any more.

The idea is to simplify the integration of mchains and agents.

The fact is that initially mchains were added to SObjectizer in order to simplify the communication of agents with other parts of the application that are written without agents. For example, there is a main application thread on which user interaction takes place using the GUI. And there are several agent-workers who perform background “hard” work. It is not a problem to send a message to the agent from the main thread: just call normal send. But how to transfer information back?

For this mchains were added.

But over time, it turned out that mchains can play a much larger role. You can, in principle, make multi-threaded applications on SObjectizer without agents at all, only on mchain (see more here ). You can also use mchains as a means of load balancing agents. As a mechanism for solving the problem of producer-consumer.

The producer-consumer problem is that if the producer generates messages faster than the consumer can process them, then we are in trouble. The message queues will grow, the performance may degrade over time, or the application will crash altogether due to memory exhaustion.

The usual solution we proposed to use in this case is to usea couple of collector-performer agents . You can also use message limits (either as the main protection mechanism or as an addition to the collector-performer). But writing collector-performer requires additional work from the programmer.

But mchains could be used for these purposes with minimal effort from the developer. So, the producer would put the next message in mchain, and the consumer would take messages from this mchain.

But the problem is that when a consumer is an agent, it is not very convenient for an agent to work with mchain with the available functions receive () and select (). And this inconvenience could be tried to eliminate with the help of some tool for the integration of agents and mchains.

When developing such a tool, you will need to solve several problems. For example, when a message arrives at mchain, at what point should it be removed from mchain? If the consumer is free and does not process anything, then you can take the message from mchain immediately and give it to the agent-consumer. If the consumer has already been sent a message from mchain, he has not yet managed to process this message, but a new message has already arrived at mchain ... How to be in this case?

There is an assumption that envelopes can help in this case. So, when we take the first message from mchain and send it to the consumer, we wrap this message in a special envelope. When the envelope sees that the message has been delivered and processed, it requests the next message from mchain (if any).

Of course, everything is not so simple. But so far it looks quite solvable. And, I hope, a similar mechanism will appear in one of the following versions of so_5_extra.

Were we going to open Pandora’s box?

It should be noted that in our case the added possibilities themselves cause ambivalent feelings.

On the one hand, the envelopes have already allowed / allowed to do things that were previously mentioned (and something was just dreamed of). For example, these are guaranteed cancellation of timers and a limit on delivery time, delivery reports, the ability to revoke a previously sent message.

On the other hand, it is not clear what this will lead to later. After all, from any opportunity you can make a problem, if you start to exploit this opportunity where necessary and where not. So can we open the Pandora's box and we ourselves still can not imagine what awaits us?

It remains only to be patient and see where all this will lead us.

On the nearest development plans of SObjectizer instead of conclusion

Instead of a conclusion, I would like to talk about how we see the nearest (and not only) future of SObjectizer. If someone doesn’t like something in our plans, you can express your opinion and influence how SObjectizer-5 will develop.

The first beta versions of SObjectizer-5.5.23 and so_5_extra-1.2.0 are already fixed and available for download and experiments. There will still be a lot of work to be done in the field of documentation and use cases. Therefore, the official release is planned in the first decade of November. If it works out earlier, we will do it earlier.

The release of SObjectizer-5.5.23, to all appearances, will mean that the evolution of branch 5.5 is coming to its final. The very first release in this thread took place four years ago, in October 2014. Since then, SObjectizer-5 has evolved within branch 5.5 without any major breaking changes between versions. It was not easy. Especially given the fact that all this time we had to look at the compilers, which was far from ideal support for C ++ 11.

Now we don’t see any sense in looking at compatibility inside the 5.5 branch and, especially, on old C ++ compilers. What could have been justified in 2014, when C ++ 14 was just getting ready to be officially adopted, and C ++ 17 was not yet on the horizon, now it looks completely different.

Plus, in SObjectizer-5.5 itself a fair amount of rakes and props have already accumulated, which appeared because of this very compatibility and which hamper the further development of SObjectizer.

Therefore, in the coming months we are going to act on the following scenario:

1. Development of the next version of so_5_extra, in which I would like to add tools to simplify the writing of tests for agents. Whether it will be so_5_extra-1.3.0 (that is, with breaking changes relative to 1.2.0) or will it be so_5_extra-1.2.1 (that is, without breaking changes) is not yet clear. Let's see how it goes. It is only clear that the next version of so_5_extra will be based on SObjectizer-5.5.

1a.If the next version of so_5_extra needs to do something extra in SObjectizer-5.5, the next version 5.5.24 will be released. If for so_5_extra it is not necessary to make improvements to the SObjectizer core, then version 5.5.23 will be the last significant version in the framework of the 5.5 branch. Minor bug-fix releases will come out. But the development of branch 5.5 itself stops at version 5.5.23 or 5.5.24.

2. Then a version of SObjectizer-5.6.0 will be released, which will open a new branch. In the 5.6 branch, we will clean the SObjectizer code from all the accumulated crutches and props, as well as from the old rubbish, which has long been marked depressed. Some things are likely to refactor (for example, abstract_message_box_t can be changed), but hardly cardinal. The basic principles of work and the characteristics of SObjectizer-5.5 in SObjectizer-5.6 will remain the same.

SObjectizer-5.6 will require C ++ 14 (at least at the level of GCC-5.5). Visual C ++ compilers below VC ++ 15 (which is from Visual Studio 2017) will not be supported.

Branch 5.6 is considered by us as a stable branch of SObjectizer, which will be relevant until the first version of SObjectizer-5.7 appears.

I would like to make release of version 5.6.0 in the beginning of 2019, approximately in February.

3. After the stabilization of the 5.6 branch, we would like to start working on the 5.7 branch, in which we could review some basic principles of SObjectizer operation. For example, completely abandon the public dispatchers, leaving only private. To alter the mechanism of cooperations and their parent-child relationships, thereby getting rid of the bottleneck during registration / deregistration of cooperations. Remove the division by message / signal. Leave only send / send_delayed / send_periodic to send messages, and hide the methods deliver_message and schedule_timer “under the hood”. Modify the message dispatch mechanism to either completely remove the dynamic_casts from this process or reduce them to the minimum.

In general, there is where to turn. In this case, SObjectizer-5.7 will already require C ++ 17, without regard to C ++ 14.

If you look at things without rose-colored glasses, then it is good if release 5.7.0 takes place at the end of autumn 2019. That is, The main working version of SObjectizer at 2019 will be the 5.6 branch.

4. In parallel, all this will develop so_5_extra. Probably, along with SObjectizer-5.6, the so_5_extra-2 version will be released, which during 2019 will incorporate a new functionality, but based on SObjectizer-5.6.

Thus, we ourselves see a progressive evolution for SObjectizer-5 with a gradual revision of some of the basic principles of SObjectizer-5. At the same time, we will try to do this as smoothly as possible so that we can switch from one version to another with minimal pain.

However, if someone would like more dramatic and significant changes from SObjectizer, then we have some thoughts on this . In short, you can remake SObjectizer as you like, to the extent that you can implement SObjectizer-6 for another programming language. But we will not do this entirely at our own expense, as it happens with the evolution of SObjectizer-5.

On this, perhaps, everything. In comments to the previous articleThere was a good and constructive discussion. It would be useful for us if a similar discussion happened this time. As always, we are ready to answer any questions, but on sensible ones, and with pleasure.

And to the most patient readers who have reached this line, thank you very much for taking the time to read the article.