The actor model is a good approach to solving some types of problems. A ready actor framework, especially in the case of C ++, can make a developer’s life much easier. The programmer is removed a lot of worries on the management of working contexts, queuing of messages, monitoring the lifetime of messages, etc. But, as they say,
all the good in this life is either illegal, or immoral, or leads to obesity for nothing. One of the problems of using a ready-made (i.e. alien) actor framework is that it sometimes turns into a black box. You see that you give into this “black box”, you see what comes from it (if it comes at all). But it is not always clear how the second is obtained from the first ...
What is it about?
One of the most common problems that developers encounter when using the
SObjectizer framework is the non-receipt of sent messages. Those. The message has been sent, but it has not reached the recipient. Why? But this is an interesting question.
There are several main reasons why a sent message does not reach the recipient:
1. The recipient simply does not exist. No longer exists or does not exist does not matter. The recipient could be when we call send, but it can cease to exist even before send completes its work. Or, calling send, we think that the recipient already exists, but in fact it has not yet been created.
')
2. The recipient did not subscribe to the message. That stupidly forgot to make a subscription to a specific message and all. When a message is sent, it is not even delivered to the agent we need, because he is simply not signed to the message. Or, alternatively, they mistakenly signed the recipient to a message from another mailbox.
3. The recipient did not subscribe to the message in the desired state. For example, the agent-turns out there are three states in which he wants to process the message. But the subscription was made only for two states, and the third was forgotten. If the receiving agent is in this third state when the message arrives, the message will not be delivered to the recipient.
There are other reasons, but these are the most common.
According to our observations, causes # 2 and # 3 are the most common. And everyone is attacking this rake, both beginners and experienced users of SObjectizer. Even ourselves, the developers of SObjectizer, regularly make these stupid mistakes. How does this happen? Yes, very simple.
Simple example with the wrong subscription
Here is the code of the simplest example in which the pinger and ponger agents exchange ping and pong signals through a shared multi-producer / multi-consumer mailbox:
#include <so_5/all.hpp> // , pinger ponger. struct ping final : public so_5::signal_t {}; struct pong final : public so_5::signal_t {}; // pinger, ping, ping pong. class pinger final : public so_5::agent_t { // , . const so_5::mbox_t mbox_; public: // . pinger(context_t ctx, so_5::mbox_t mbox) : so_5::agent_t{std::move(ctx)} , mbox_{std::move(mbox)} { // . so_subscribe(mbox_).event([this](mhood_t<pong>) { std::cout << "pong!" << std::endl; so_5::send<ping>(mbox_); }); } // ping. virtual void so_evt_start() override { so_5::send<ping>(mbox_); } }; // ponger, pong- ping-. class ponger final : public so_5::agent_t { // , . const so_5::mbox_t mbox_; public: // . ponger(context_t ctx, so_5::mbox_t mbox) : so_5::agent_t{std::move(ctx)} , mbox_{std::move(mbox)} { // . so_subscribe_self().event([this](mhood_t<ping>) { std::cout << "ping!" << std::endl; so_5::send<pong>(mbox_); }); } }; int main() { // SObjectizer . so_5::launch([](so_5::environment_t & env) { env.introduce_coop([](so_5::coop_t & coop) { // , . const auto mbox = coop.environment().create_mbox(); // . coop.make_agent<pinger>(mbox); coop.make_agent<ponger>(mbox); }); }); }
There is a small error in this simple code, due to which, when running the example, we will not see the printing of ping / pong messages. Perhaps someone already understood what was happening. But, generally speaking, it is not so easy to detect such annoying errors. Especially when you deal not with examples,
but with real agents , the logic and implementation of which can be much more complicated.
The effect of "black box" and what to do with it?
In the example shown above, we are faced with the fact that SObjectizer for us has become a “black box”. We give out some control actions to him, but the result we need does not happen. But why? What is the reason? How to get to the bottom of this reason?
And here it turns out that getting to the bottom of the cause is not so easy. We need to somehow debug our small example, but how to do it?
If we use debuggers, then we can put breakpoints in several places. For example:
1. In the pinger :: so_evt_start () method to make sure that the first send for the ping is being called.
2. In the ping signal handler in the ponger agent, in order to make sure that the first ping arrives and the pong is sent in response.
Or, if we are real programmers and we believe that debuggers are for weaklings, then we can arrange debug seals in the same places.
And having done these actions, we find that so_evt_start () is called, the first send for ping works, but the handler for pong is not called. But why? We made a subscription in the constructor ...
Here, just before the developer who uses SObjectizer, the question arises: “But how can you look inside SObjectizer to understand exactly how the message is delivered?”
The question is not the easiest. Because, if the programmer decides to send a send call, he will first get into the jungle of the details of the SObjectizer implementation and is unlikely to enjoy it (although some looked and said that they didn’t see anything wrong, but could they believe?). Secondly, and most importantly, he will understand that message delivery consists of two parts. The first part is setting up the message in the order queue for subscribers. The second part is the maintenance of a specific application, which is retrieved from the queue by a specific dispatcher.
And well, if the problem is found in the first part. For example, the developer will see that there are no subscribers for the message and therefore no requests are generated for the message. But if the applications were generated, but then the handler was not called, then the programmer will have to dive deeper into the giblets of SObjectizer. And there he can find out that there are different dispatchers in SObjectizer, some very specific ones, who work differently than others, and it’s not so easy to find the place where the next application is extracted and analyzed from the queue.
In general, if someone wants a debugger to go through the message delivery mechanism in SObjectizer, then this person can only wish good luck. Since we ourselves are not big fans of this pastime.
Is there any alternative?
There is. This is a
mechanism that is officially called message delivery tracing (or msg_tracing for simplicity). It was added to SObjectizer-5 two and a half years ago. But, probably, few people have heard about him, although we ourselves use it constantly.
Mechanism msg_tracing
When you run SObjectizer, you can enable the msg_tracing mechanism. If it is enabled, then SObjectizer will generate text messages describing the process of delivering and processing messages. These messages will be passed to a special tracer object. The task of the tracer is to save or some other processing of trace messages. The user can either implement his own tracer himself, or he can use one of the ready tracer from SObjectizer.
Let's extend the example shown above by turning on msg_tracing and see what happens. So, turn on msg_tracing and specify that all trace messages should be displayed on the standard output stream:
int main() {
Run the modified example and get something like:
[tid = 13728] [mbox_id = 5] deliver_message.no_subscribers [msg_type = struct ping] [signal] [overlimit_deep = 1]
This entry indicates that the deliver_message operation is in progress. At the same time, a situation was discovered when there are no subscribers for the ping signal sent to the mailbox with such an identifier.
So, we see that the message is sent to the mailbox, but nobody has subscribed to it. But as so, we made a subscription, here it is:
And here we can already see that we did a subscription, but not at all on that mbox. We needed to call so_subscribe (mbox_), and we called so_subscribe_self (). An annoying mistake that is made with enviable regularity. The msg_tracing mechanism allowed us to quickly deal with the problem, without diving into the SObjectizer giblets with the debugger.
msg_tracing is a debugging mechanism
The most important thing to remember is that msg_tracing is a debugging, auxiliary mechanism. It was created to help developers with debugging applications developed on the basis of SObjectizer.
In particular, additional overhead costs are associated with msg_tracing, which significantly increases the cost of sending and receiving messages. That is why msg_tracing needs to be included explicitly, this is done only once and only before starting the SObjectizer. You cannot disable msg_tracing after SObjectizer has been started with msg_tracing turned on. Everything is so serious that with msg_tracing enabled, other data structures containing both additional data and additional code are used inside the SObjectizer. So, inside the SObjectizer, when msg_tracing is turned on, other types of mboxes are created.
Therefore, msg_tracing should be considered as an auxiliary tool for debugging SObjectizer applications. And you should not build on its base any means that will have to work in production.
The current state of the msg_tracing mechanism: its disadvantages and possible solutions
The msg_tracing mechanism was added to version 5.5.9 in October 2015. Even then it was clear that this mechanism itself is in its infancy. It was necessary to start with something, it was necessary to get something working that one could try in practice, gain experience and understand where to go next.
Since that time, two serious miscalculations / deficiencies have been identified in the existing msg_tracing mechanism.
No trace filtering
If msg_tracing was enabled at the start of SObjectizer, then SObjectizer created text trace messages for all operations related to message delivery and passed these messages to the tracer object. This is not a problem when msg_tracing is used to debug a small application, example, or test. But if you try to use msg_tracing in a large application with hundreds of agents inside that exchange millions of messages, this is a problem. Since you need to somehow filter out the unnecessary programmer trace messages, and doing this with text trace messages is not so easy.
Correspondingly, from time to time, there was talk of allowing the SObjectizer user to filter trace messages before they are transmitted to the tracer. Moreover, the filter must deal not with the finalized text message, but with the data that is then converted into text.
It is not possible to control the flow of trace messages.
Sometimes I would like to launch a SObjectizer with blocked trace messages, and then, at some point in time, “open the tap” and allow trace messages to get into the tracer. And then "turn off the tap."
Accordingly, from time to time, there was talk that msg_tracing could be turned on and off during the operation of SObjectizer. Although the difficulty here was that at its launch, SObjectizer should already know that the developer wants to have msg_tracing. In this case, the SObjectizer will create other types of mboxes and will use other methods of message delivery (with a trace of what is happening).
Upcoming modification msg_tracing in version 5.5.22
The other day we recorded the first alpha of the new version of SObjectizer. It has an extension of the msg_tracing mechanism. The concept of a filter for trace messages has been added.
Filter is optional. If a filter is assigned, before forming a text trace message, SObjectizer first asks the filter if it is necessary to allow further processing of the current trace message. If the filter says that “yes, it is possible”, then a text trace message is generated and it is given to the tracer object. If the filter says “no, not”, then the trace message is not generated and does not go anywhere.
If there is no filter, then SObjectizer works as before: a text trace message is generated and sent to the tracer object. Those. if the programmer does not know about filters or does not want to use filters, then the msg_tracing mechanism works as before for him.
In addition, SObjectizer in version 5.5.22 allows you to set and delete msg_tracing-filters during their work.
An example of using msg_tracing-filters in version 5.5.22
To demonstrate the new features of the msg_tracing mechanism, let's take the following example. There are two agents that work almost equally. They have three states: first, second and third. In each of their states, agents must respond to two types of messages. According to the change_state message, you need to go to the next state (ie, from first to second, from second to third, from third to first). If the tick message arrives, then the agent needs to simply react to it. What will be the reaction - it does not matter.
The difference between the first and second agents will be that the first agent responds to the tick message in each of its states. But the second agent subscribed to the message tick in the second state forgot. Accordingly, the second agent will lose tick messages when it is in the second state. It is precisely these losses that we want to detect using the msg_tracing mechanism.
To do this, we need to install a trace filter of the following form:
It looks probably scary. But let's try to figure it out.
The change_message_delivery_tracer_filter () method allows you to replace the trace filter during SObjectizer operation.
The auxiliary function make_filter creates an object of the desired SObjectizer type from a lambda function.
A single argument is passed to this lambda function — a reference to the trace_data_t interface, from which the filter can extract trace-related data. In this particular case, we call trace_data_t the only method event_handler_data_ptr (). This method returns optional <const event_handler_data *>. Accordingly, inside optional can either be a pointer or not. If there is no pointer, then the trace message means that we are not interested, since it does not relate to the search for a message handler. If there is a pointer, but it is null, then this is exactly the case that we are waiting for: SObjectizer tried to find a handler for the message, but found nothing. In this and only in this case, we allow the trace message.
If we run this example, we see that the following messages appear on the console periodically:
[tid = 11880] [agent_ptr = 0x23be91d7cb0] demand_handler_on_message.find_handler [mbox_id = 6] [msg_type = struct base :: tick] [signal] [state = second] [evt_handler = NONE]
[tid = 11880] [agent_ptr = 0x23be91d7cb0] demand_handler_on_message.find_handler [mbox_id = 6] [msg_type = struct base :: tick] [signal] [state = second] [evt_handler = NONE]
[tid = 11880] [agent_ptr = 0x23be91d7cb0] demand_handler_on_message.find_handler [mbox_id = 6] [msg_type = struct base :: tick] [signal] [state = second] [evt_handler = NONE]
[tid = 11880] [agent_ptr = 0x23be91d7cb0] demand_handler_on_message.find_handler [mbox_id = 6] [msg_type = struct base :: tick] [signal] [state = second] [evt_handler = NONE]
And they are then baked, then not printed. This is because we have a trace_controller agent, which then "turns on" the stream of trace messages, then "turns off" it.
For those interested here is the full code of the example. #include <so_5/all.hpp> // . class base : public so_5::agent_t { protected: // , // . struct tick final : public so_5::signal_t {}; // . struct change_state final : public so_5::signal_t {}; // , . state_t st_first{this, "first"}, st_second{this, "second"}, st_third{this, "third"}; // . , // . so_5::timer_id_t tick_timer_; so_5::timer_id_t change_state_timer_; public: base(context_t ctx) : so_5::agent_t{std::move(ctx)} {} // . virtual void so_define_agent() override { // . this >>= st_first; // change_state . st_first.event([this](mhood_t<change_state>) { this >>= st_second; }); st_second.event([this](mhood_t<change_state>) { this >>= st_third; }); st_third.event([this](mhood_t<change_state>) { this >>= st_first; }); } // , . virtual void so_evt_start() override { using namespace std::chrono_literals; // 250ms. change_state_timer_ = so_5::send_periodic<change_state>(*this, 250ms, 250ms); // tick. tick_timer_ = so_5::send_periodic<tick>(*this, 0ms, 100ms); } }; // , tick . class first_agent final : public base { public: using base::base; virtual void so_define_agent() override { base::so_define_agent(); // . so_subscribe_self() .in(st_first).in(st_second).in(st_third) .event([](mhood_t<tick>){}); } }; // , tick st_second. class second_agent final : public base { public: using base::base; virtual void so_define_agent() override { base::so_define_agent(); // . so_subscribe_self() .in(st_first).in(st_third) .event([](mhood_t<tick>){}); } }; // , "" "" // trace-. class trace_controller final : public so_5::agent_t { // /. struct on_off final : public so_5::signal_t {}; // , . state_t st_on{this}, st_off{this}; // on_off. so_5::timer_id_t timer_; public: trace_controller(context_t ctx) : so_5::agent_t{std::move(ctx)} { st_off.event([this](mhood_t<on_off>) { // . this >>= st_on; // , // - . , // . so_environment().change_message_delivery_tracer_filter( so_5::msg_tracing::make_filter( [](const so_5::msg_tracing::trace_data_t & td) { // . const auto handler_ptr = td.event_handler_data_ptr(); if(handler_ptr) { // trace- . // , . if(nullptr == *handler_ptr) // . , // . Trace- . return true; } // trace-. return false; })); }); st_on.event([this](mhood_t<on_off>) { // . this >>= st_off; // . so_environment().change_message_delivery_tracer_filter( // trace-. so_5::msg_tracing::make_disable_all_filter()); }); this >>= st_off; } virtual void so_evt_start() override { using namespace std::chrono_literals; timer_ = so_5::send_periodic<on_off>(*this, 0ms, 1500ms); } }; int main() { // SObjectizer . so_5::launch([](so_5::environment_t & env) { env.introduce_coop([](so_5::coop_t & coop) { // // first_agent second_agent. coop.make_agent<first_agent>(); coop.make_agent<second_agent>(); // , // trace-. coop.make_agent<trace_controller>(); }); }, [](so_5::environment_params_t & params) { // . // . params.message_delivery_tracer(so_5::msg_tracing::std_cout_tracer()); // trace- , . params.message_delivery_tracer_filter( so_5::msg_tracing::make_disable_all_filter()); }); }
What is available now via trace_data_t?
Currently, the trace_data_t interface is defined as follows:
class trace_data_t { ... public :
It should be especially emphasized that all this information will not always be available for the trace message. For example, if a message is sent to a mailbox for distribution to subscribers, then event_handler_data_ptr () will return an empty optional. If the mailbox does not have subscribers for this type of message, then the agent () method will return an empty optional.
Why was all this described?
This article was written in order to enable those interested developers to know in advance about what awaits them in the next version of SObjectizer.
Accordingly, they can familiarize themselves with the proposed innovations, can feel them, make their impression and express their “Phi”, if they don’t like something ( so-5.5.22-alpha1 is available for download, the mirror on github is also updated ).Well, we, accordingly, will have the opportunity to understand what and how to improve even before fixing the final version 5.5.22. For us, this is important, because we have a leap on the theme of maintaining compatibility between versions and we really do not like to break this very compatibility.So if someone is interested in this topic, then please speak in the comments. We will try to listen to constructive considerations.Ps. , ,
.
PPS. 5.5.22
. , , , , 5.5.22 .