Our approach to coloring the flow

We in the company always strive to increase the maintainability of our code using generally accepted practices, including in matters of multithreading. This does not solve all the difficulties that the ever-increasing workload brings, but it simplifies the support - the readability of the code and the speed of developing new features gains.

We now have 47,000 users daily, about 30 servers in production, 2,000 API requests per second and daily releases. The Miro service has been developing since 2011, and in the current implementation, user requests are processed in parallel by a cluster of heterogeneous servers.

Competitive Access Management Subsystem

The main value of our product - collaborative custom boards, so the main burden falls on them. The main subsystem that controls most of the competitive access is the stateful system of user sessions on the board.
')
For each board being opened on one of the servers, a status is raised. It stores both application runtime data required for collaboration and content display, as well as system data, such as binding to processing threads. Information on which server the state is stored in is written into the distributed structure and is available to the cluster as long as the server is running and at least one user is on the board. We use Hazelcast to make this part of the subsystem work. All new connections to the board are sent to the server with this status.

When connecting to the server, the user enters the receiving stream, the only task of which is to bind the connection to the state of the corresponding board, in the streams of which all further work will occur.

There are two threads associated with the board: the network, processing connections, and the “business” one, which is responsible for the business logic. This allows you to turn the performance of heterogeneous tasks of processing network packets and executing business commands from serial to parallel. Processed network commands from users form applied business tasks and direct them to the business flow, where they are processed sequentially. This allows you to avoid unnecessary synchronization in the development of application code.

Dividing the code into business / application and system is our internal convention. It allows you to distinguish between the code responsible for the features and capabilities for users, from the low-level details of communication, sheduling and storage, which are serving tools.

If the receiving thread detects that there is no state for the board, the corresponding initialization task is set. The initialization of the state is handled by a separate stream type.

Task types and their direction can be represented as follows:



This implementation allows us to solve the following problems:

1. There is no business logic in the receiving thread that could slow down a new connection. This type of stream on the server exists in a single copy, so the delays in it immediately affect the time of opening the boards, and if an error occurs in the business code, it can be easily hung up.
2. The state initialization is not performed in the business flow of the boards and does not affect the processing time of business commands from users. It may take some time, and business streams process several boards at once, so the opening of new boards does not directly affect already working ones.
3. Network command parsing is often faster than direct execution, so the configuration of a pool of network streams may be different from the configuration of a pool of business flows in order to use system resources efficiently.

Coloring pages

The subsystem described above in the implementation is quite nontrivial. The developer has to keep in mind the scheme of the system and take into account the reverse process of closing the boards. When closing, it is necessary to remove all subscriptions, delete entries from the registry and do it in the same threads in which they were initialized.

We noticed that the bugs and complexities of code modification that arose in this subsystem were often associated with a lack of understanding of the execution context. Juggling with threads and tasks made it difficult to answer the question in which thread a particular code section was executed.

To solve this problem, we used the flow coloring method — this is a policy aimed at regulating the use of flows in the system. Threads are assigned colors, and methods define the framework for execution within threads. The color here is an abstraction, it can be any entity, for example, an enumeration. In Java, annotations can serve as color markup:

@Color @IncompatibleColors @AnyColor @Grant @Revoke 

Annotations are added to the method, using them you can set the validity of the method. For example, if the annotation of the method allows yellow and red color, then the first stream can call the method, and for the second such call will be erroneous.



You can specify invalid colors:



You can dynamically add and remove thread privileges:



The absence of annotation or annotation as in the example below says that the method can run in any stream:



Android developers may be familiar with this approach from the annotations of MainThread, UiThread, WorkerThread, etc.

In the coloring of streams, the principle of self-documentation of the code is used, and the method itself lends itself well to static analysis. Using static analysis, you can say before the execution of the code whether it was written correctly or not. If we exclude the Grant and Revoke annotations and assume that the stream already has an immutable set of privileges during initialization, this will be a flow-insensitive analysis — a simple version of static analysis that does not take into account the order of calls.

At the time of introducing the method of coloring streams in our devops-infrastructure there were no ready-made solutions for static analysis, so we went in a simpler and cheaper way - we introduced our own annotations, which are uniquely associated with each type of streams. We began to check their correctness using aspects in runtime.

 @Aspect public class ThreadAnnotationAspect { @Pointcut("if()") public static boolean isActive() { … //   ,    . , ,    } @Pointcut("execution(@ThreadAnnotation * *.*(..))") public static void annotatedMethod() { } @Around("isActive() && annotatedMethod()") public Object around(ProceedingJoinPoint joinPoint) throws Throwable { Thread thread = Thread.currentThread(); Method method = ((MethodSignature) jp.getSignature()).getMethod(); ThreadAnnotation annotation = getThreadAnnotation(method); if (!annotationMatches(annotation, thread)) { throw new ThreadAnnotationMismatchException(method, thread); } return jp.proceed(); } } 

For aspects, we use the aspectj library and the maven plugin, which weaving while compiling the project. Initially weaving was configured to load-time when loading classes with a ClassLoader. However, we were faced with the fact that weaver sometimes incorrectly behaved with competitive loading of the same class, as a result of which the source code of the class remained unchanged. In the end, this resulted in a very unpredictable and difficult to reproduce behavior in production. Perhaps in the current versions of the library there is no such problem.

The solution on aspects allowed us to quickly find most problems in the code.

It is important not to forget to always keep the annotations up to date: you can delete them, add some laziness, weaving aspects can be turned off altogether - in this case the coloring will quickly lose its relevance and value.

GuardedBy

One of the varieties of coloring includes the abstract GuardedBy from java.util.concurrent. It differentiates access to fields and methods, indicating which locks are necessary for correct access.

 public class PrivateLock { private final Object lock = Object(); @GuardedBy (“lock”) Widget widget; void method() { synchronized (lock) { //Access or modify the state of widget } } } 

Modern IDEs even support the analysis of this annotation. For example, IDEA gives the following message if something is wrong in the code:


The threading method itself is not new, but it seems that in languages ​​such as Java, where often multi-threaded access goes to mutable objects, using it not only as part of documentation, but also at the compilation stage, assembly could greatly simplify the development of multi-threaded code.

We still use implementation on aspects. If you are familiar with a more elegant solution or analysis tool that improves the resilience of this approach to changing the system, please share it in the comments.