Building blocks distributed applications. Zero approximation

The world does not stand still. Progress creates new technological challenges. In accordance with the changing requirements, the architecture of information systems must evolve. Today we will talk about event-oriented architecture, competitiveness, parallelism, asynchrony and how to live in Erlang peacefully with all this.

Introduction

Depending on the size of the designed system and the requirements for it, we, the developers, choose the method of information exchange in the system. In most cases, for organizing the interaction of services, a working variant can be a scheme with a broker, for example, based on RabbitMQ or kafka. But sometimes the flow of events, SLA and the level of control over the system are such that ready-made messaging does not suit us. Of course, you can complicate the system a little by taking responsibility for the transport layer and cluster formation, for example using ZeroMQ or nanomsg. But if the system has enough bandwidth and capabilities of a standard Erlang cluster, then the issue of adding an additional entity requires detailed study and economic justification.

The topic of reactive distributed applications is quite extensive. To fit into the format of the article, the subject of today's discussion will be only homogeneous media built on the basis of Erlang / Elixir. Ecosystem Erlang / OTP allows you to implement a reactive architecture with the lowest labor costs. But in any case, we need a messaging layer.

Theoretical basis

Design begins with defining goals and constraints. The main goal is not in development for the sake of development. We need to get a safe and scalable tool on the basis of which we can create, and most importantly, develop modern applications of different levels: from single-server servers to a small audience that can later develop into clusters of up to 50-60 nodes, ending with cluster federations. Thus, the main goal is to maximize profits by reducing the cost of developing and owning the final system.

There are 4 main requirements for the final system:

• With everyday orientation.
  
  The system is always ready to pass a stream of events through itself and perform the necessary actions;
• M scalability.
  
  Individual blocks can be scaled both vertically and horizontally. The whole system must have the possibility of infinite horizontal growth;
• About durability.
  
  All levels and all services should be able to automatically recover from failures;
• Guaranteed response time.
  
  Time is valuable and users should not wait too long.

Do you remember the old tale about “The little engine that could”, he is “The engine that could”? In order for the projected system to successfully exit the prototype stage and be progressive, its foundation must meet the minimum requirements of SMOG .

One more point is added to messaging as an infrastructure tool and a basis for all services: usability for programmers.

Event orientation

In order for an application to grow from one server to a cluster, its architecture must provide weak connectivity. This requirement is met by an asynchronous model. In it, the sender and recipient care about the information load of the message and do not worry about the transmission and routing within the system.

Scalability

The scalability and efficiency of the system stand nearby. Application components should be able to dispose of all available resources. The more efficiently we can utilize power and the more optimal our processing methods, the less we spend money on equipment.

Within the same machine, Erlang creates a highly competitive environment. The balance between concurrency and parallelism can be defined by choosing the number of operating system threads available for the Erlang VM and the number of schedulers that recycle these threads.

Erlang processes have no common state and work in non-blocking mode. This provides a relatively low latency and higher throughput than traditional applications built on blocking synchronization. The Erlang scheduler takes care of fair allocation of CPU and IO, and the absence of locks allows the application to respond even in peak loads or failures.

At the cluster level, the problem with recycling also exists. It is important that all the machines in the cluster are uniformly loaded, and the network is not overloaded. Imagine a situation: user traffic lands on incoming balancers (haproxy, nginx, etc), they distribute processing requests as evenly as possible between the set of available backends. Within the application infrastructure, a service that implements the requested interface is only the last mile, and it will need to request a number of other services to respond to the initial request. Internal requests also require routing and balancing.

In order to effectively manage data flows, messaging must provide developers with an interface to manage routing and load distribution. Due to this, developers can, using microservice patterns (aggregator, proxy, chain, branch, etc), solve both standard tasks and rarely occurring ones.

From a business perspective, scalability is one of the risk management tools. The main thing is to satisfy clients' requests, using the equipment optimally:

• With the increase in power equipment as a result of progress. It will not be idle due to software imperfections. Erlang vertically scales perfectly and can always utilize all the CPU cores and available memory;
• In cloudy environments, we can control the amount of equipment depending on the current or projected load and guarantee the SLA.

fault tolerance

Consider two axioms: “Failures are unacceptable” and “Failures will always be.” For a business, software failure is a loss of money, and worse, a reputation. Balancing between potential losses and the cost of developing fault-tolerant software, you can often find a compromise.

In the short term, the architecture in which resiliency is incorporated saves money on the purchase of ready-made clustering solutions. They are expensive, and they also have errors.

In the long-term, fault-tolerant architecture repeatedly pays for the costs of its application at all stages of development.

Messaging within the code base at the development stage allows for a detailed study of the interaction of components within the system. This simplifies the task of responding and managing failures, since all the critical components handle failures, and the final system knows how to automatically return to a normal state after a failure by design.

Responsiveness

Regardless of failures, the application must respond to requests and satisfy the SLA. The reality is that people do not want to wait, respectively, the business must adapt. From a growing number of applications expect high responsiveness.

Responsive applications work in real-time mode. Erlang VM operates in soft real-time mode. For some areas, such as stock trading, medicine, industrial equipment management, hard real-time is important.

Responsive systems improve UX and are useful for business.

Preliminary result

Planning this article, I wanted to share the experience of creating a messaging broker and building complex systems on its basis. But the theoretical and motivational part turned out quite extensive.

In the second part of the article, I will discuss the nuances of implementing exchange points, message exchange patterns and their application.

In the third part we consider the general issues of the organization of services, routing and balancing. Let's talk about the practical side of scalability and fault tolerance systems.

The end of the first part.

Photos by @ lucabravo .