From the experience of using SObjectizer: are the actors in the form of finite automata - is it bad or good?

Having acquainted readers with the SObjectizer framework , its capabilities and features, you can proceed to the story of some of the lessons that we have learned in more than fourteen years of using SObjectizer in developing C ++ software. Today we will talk about when agents in the form of finite automata are not a good choice, but when they are. The fact that the ability to create a large number of agents is not so much a solution as a problem in itself. And how the first relates to the second ...

So, in the three previous articles ( one , two, and three ), we watched the email_analyzer agent evolve from a very simple to a more or less complex class. I think that many who looked at the final version of email_analyzer , the question arose: “But this is very difficult, could it not be easier?”

It turned out so difficult because the agents are represented as finite automata. In order to process an incoming message, a separate method must be described - the event handler. In order for the agent to start a new event handler, the current handler must complete. Therefore, in order to send a request and receive a response, the agent must complete its current handler in order to give the dispatcher the ability to call the appropriate handler when a response is received. Those. instead:

void some_agent::some_event() { ... //  . send< request >(receiver, reply_to, params…); //     . auto resp = wait_reply< response >(reply_to); ... //  . } 

You have to write like this:

 void some_agent::some_event() { ... //       . so_subscribe(reply_to).event(&some_agent::on_response); //  . send< request >(receiver, reply_to, params...); //    some_event  . //         //  ,   . } void some_agent::on_response(const response & resp) { ... //  . } 

From here both such volume, and such complexity at the resulting agent email_analyzer.

It is possible that in this approach there will be some tricks that would reduce the volume of writings by 20-30%, but the situation will not change fundamentally.

What can significantly affect the comprehensibility and compactness is the departure from the event model based on callbacks in the direction of a linear code with synchronous operations. Sort of:

 void email_analyzer(context_t ctx, string email_file, mbox_t reply_to) { try { //   . auto raw_content = request_value< load_email_succeed, load_email_request >( ctx.environment().create_mbox( "io_agent" ), 1500ms, //     1.5s email_file ).content_; auto parsed_data = parse_email( raw_content ); //  -checker-,     //   message chain,     . auto check_results = create_mchain( ctx.environment() ); introduce_child_coop( ctx, disp::thread_pool::create_disp_binder( "checkers", disp::thread_pool::bind_params_t{} ), [&]( coop_t & coop ) { coop.make_agent< email_headers_checker >( check_results, parsed_data->headers() ); coop.make_agent< email_body_checker >( check_results, parsed_data->body() ); coop.make_agent< email_attach_checker >( check_results, parsed_data->attachments() ); } ); // ..      ,   //      . auto check_handler = [&]( const auto & result ) { if( check_status::safe != result.status ) throw runtime_error( "check failed: " + result ); } ); //     0.75s   ,  //      . auto r = receive( from( check_results ).total_time( 750ms ), [&]( const email_headers_check_result & msg ) { check_handler( msg ); }, [&]( const email_body_check_result & msg ) { check_handler( msg ); }, [&]( const email_attach_check_result & msg ) { check_handler( msg ); } ); //     ,    . if( 3 != r.handled() ) throw runtime_error( "check timedout" ); //      ,    . send< check_result >( reply_to, email_file, check_status::safe ); } catch( const exception & ) { send< check_result >( reply_to, email_file, check_status::check_failure ); } } 

In such a case, a more compact and understandable code would turn out that would be similar to the solution of this problem in languages ​​such as Erlang or Go.

Our experience suggests that in situations where an agent performs some linear set of operations like “sent a request, immediately began to wait for a single answer”, its implementation in the form of a finite state machine will be disadvantageous in terms of code size and complexity. Instead of simply waiting for a response and immediately continuing to work after receiving it, the agent needs to complete its current event handler, and all other actions have to be moved to another handler. If an agent performs N consecutive asynchronous operations during its life, then this agent is likely to have (N + 1) handler. What is not good, because the development and maintenance of such an agent will take a lot of time and effort.

The situation will be completely different if at each moment when the agent is waiting for something, several different messages may come to him, and the agent will have to react to each of them. For example, an agent may wait for the result of the current operation, and at that moment requests may come to the agent to check the status of the operation and request to perform a new operation. In this case, at each agent waiting, you will have to paint a reaction to all the expected types of messages, and this can also quite quickly turn the agent code into bulky and poorly understood noodles.

Since the SObjectizer currently only supports agents in the form of finite automata, it is necessary to carefully evaluate how well the logic of the applied agents lies on the finite automata. If not very well, then SObjectizer may not be the best choice and it makes sense to look at solutions that use coroutines. For example, boost.fiber or Synca (about the last were interesting articles on Habré: №1 and №2 ).

So the three previous articles in the SObjectizer miniseries: from simple to complex, on the one hand, show the capabilities of SObjectizer, but, on the other hand, allow you to see where you can go, there is a solution to the problem from the wrong side. For example, if you start using agents in the form of finite automata where it would make sense to use agents in the form of coroutines.

But if for many cases coroutines are more profitable than finite automata, then why does SObjectizer not support agents in the form of coroutines? There are several serious reasons for this, both technical and organizational. Probably, if coroutines were part of the C ++ language, coroutines in SObjectizer would already be. But since Coroutines in C ++ are now available only through third-party libraries and the topic is not the easiest, then we are not in a hurry with adding this functionality to SObjectizer. Moreover, this problem has a completely different side. But to talk about it, you need to go from afar ...

A long time ago, when the first version of SObjectizer was launched, we ourselves made the same mistake as many newbies who first got a tool based on the model of actors: if you can create agents for each other, then you need to create. The execution of any task should be presented in the form of an agent. Even if this task is to receive only one request and send only one answer. In general, intoxication from new opportunities because of what suddenly you begin to hold the opinion that "there is nothing in the world except agents."

This resulted in several negative consequences.

First, the application code turned out to be larger and more complex than we would like. After all, asynchronous messages are prone to losses and, where one could write one synchronous call, there were bells and whistles around sending a request message, processing a reply message, counting a time-out to diagnose a loss of a request or response. When analyzing the code, it turned out that somewhere in half of the cases the interaction on the messages is justified, since there was a transfer of data between different workflows. And in the remaining places it was possible to merge a bunch of small agents into one big one and carry out all the operations inside him through normal synchronous function calls.

Secondly, it turned out that the behavior of an application built on agents is much more difficult to follow and even more difficult to predict. A good analogy would be to observe the flight of a large flock of birds: although the rules of behavior of an individual are simple and clear, it is almost impossible to predict the behavior of the entire flock. So it is in an application in which tens of thousands of agents live at the same time: each of them works in an understandable way, but the cumulative effect of their joint work can be unpredictable.

What else is bad is an increase in the amount of information that is needed to understand what is happening in the application. Take our example with email_analyzer. A single analyzer_manager agent can supply information such as the total number of requests waiting for its queue, the total number of live agents email_analyzer, and the minimum, maximum, and average waiting times for a request in the queue (the same for requests processing times). Therefore, monitoring the activities of analyzer_manager is not a problem. But the collection, aggregation and processing of information from individual email_analyzer-s is already more difficult. Moreover, the more difficult, the more of these agents and the shorter their lifetime.

So, the fewer agents live in an application, the easier it is to keep track of them, the easier it is to understand what is happening and how, the easier it is to predict the behavior of the application in certain conditions.

Thirdly, unpredictability, which occurs from time to time in applications with tens of thousands of agents inside, can cause the application to be partially or completely inoperable.

A characteristic case: in the application under a hundred thousand agents. They all use periodic messages to control the timeouts of their operations. And then at one point, time-outs immediately come for, say, 20 thousand agents. Correspondingly, on the worker threads, the message queues for processing are swelling. These queues begin to rake, each agent receives its message and processes it. But while these 20 thousand messages are being processed, too much time passes and another 20 thousand arrive from the timer. This is in the appendix to the part of the old messages that are still in queues. It is clear that everything does not have time to be processed and another 20 thousand messages arrive. Etc. The application seems to be honestly trying to work, but gradually degrades to complete inoperability.

As a result of walking on these rakes at the very beginning of using SObjectizer in our projects, we came to the conclusion that the ability to create a million agents is more of a marketing bullet than a thing demanded in our practice * . And that the approach, which became known as SEDA-way , allows you to build applications that are much easier to control and that behave much more predictably.

The essence of using the SEDA approach in conjunction with the model of actors is that instead of creating actors who perform a whole chain of successive operations, it is better to create one actor for each operation and build them into a pipeline. For our example with email analyzers, instead of doing email_analyzer agents that sequentially load email content, parse and analyze this content, we could do several stage agents. One stage agent would control the queue of requests. The next stage agent would handle file download operations with email. The next stage agent would parse the loaded content. The next is analysis. Etc.

The key point is that in the previously shown implementations, email_analyzer itself initiates all operations, but only for one specific email. And in the SEDA approach, we would have one agent for each operation, but each agent could do it for several emails at once. By the way, the traces of this SEDA approach are visible even in our examples as an IO agent, which is nothing but a stage agent from SEDA.

And so, when we began to actively use ideas from SEDA, it turned out that stage agents are quite conveniently implemented as finite automata, since they at each particular moment of time have to expect different incoming actions and react to them depending on their state. Here, in our opinion, in the long run, finite automata are more convenient than coroutines.

By the way, one more thing can be noted that is often paid attention to by those who first meet with SObjectizer for the first time: verbosity of agents. Indeed, as a rule, an agent in SObjectizer is a separate C ++ class that, at a minimum, has a constructor, will have some fields that need to be initialized in the constructor, the so_define_agent () method will be redefined, several event handlers will be defined as methods ... It is clear that for simple cases all this leads to a fair syntactic overhead (c). For example, in Just :: Thread Pro, a simple actor logger might look like this:

 ofstream log_file("..."); actor logger_actor( [&log_file] { for(;;) { actor::receive().match<std::string>([&](std::string s) { log_file << s << endl; } ); } } ); 

Whereas in SObjectizer, if you use the traditional approach to writing agents, you will need to do something like:

 class logger_actor : public agent_t { public : logger_actor( context_t ctx, ostream & stream ) : agent_t{ctx}, stream_{stream} {} virtual void so_define_agent() override { so_subscribe_self().event( &logger_actor::on_message ); } private : ostream & stream_; void on_message( const std::string & s ) { stream_ << s << endl; } }; ... ofstream log_file("..."); env.introduce_coop( [&log_file]( coop_t & coop ) { coop.make_agent< logger_actor >( log_file ); } ); 

Obviously, in SObjectizer scribbling more. However, the paradox is that if you follow the SEDA approach, when there are not very many agents, but they can process different types of messages, the agent code swells quite quickly. Partly because of the very logic of the agents (as a rule, more complex), partly due to the fact that agents are filled with additional things, such as logging and monitoring. And it turns out that when the main application code of an agent has a volume of several hundred lines, or even more, the size of the syntactic overhead by SObjectizer is completely insignificant. Moreover, the larger and more complex the agent, the more profitable is its representation in the form of a separate C ++ class. In toy examples, this is not visible, but in the “combat” code one feels quite strongly (well, let's say, a small example of a not very complicated real agent).

Thus, on the basis of our practical experience, we came to the conclusion that, if we properly combine the model of actors and the SEDA approach, then the representation of agents in the form of finite automata is quite a normal solution. Of course, somewhere such a decision will lose coroutines in expressiveness. But in general, agents in the form of finite automata work more than well and do not create any special problems. Except, perhaps, comparisons of various approaches to the implementation of the model of actors on micro-measures .

At the end of the article I want to appeal to readers. We have another article in our plans, in which we want to touch on such an important problem of the mechanism of interaction based on asynchronous messages, like agent overloading. And at the same time, show how SObjectizer reacts to errors in agents. But it would be interesting to know the opinion of the audience: what did you like, what did not like, what I would like to know more about. This will greatly help us both in the preparation of the next article and in the development of the SObjectizer itself.

---

* We emphasize that we are talking about our experience. It is obvious that other teams solving other tasks using the model of actors can successfully use a large number of actors in their applications. And it has the exact opposite opinion.