Orchestrated saga or how to build business transactions in services with the pattern database per service

Hello! My name is Konstantin Evteev, I work in Avito as the head of the DBA unit. Our team develops Avito data storage systems, helps in selecting or issuing databases and related infrastructure, supports Service Level Objective for database servers, and we are also responsible for efficient use of resources and monitoring, advising on design, and possibly developing microservices, tied to storage systems or services for the development of the platform in the context of storage.

I want to tell how we decided one of the challenges of the microservice architecture - carrying out business transactions in the infrastructure of services built using the Database per service pattern. I spoke on this topic at the Highload ++ Siberia 2018 conference.

Theory. As briefly as possible

I will not describe in detail the theory of sagas. I will give only a brief introductory, so that you understand the context.

As it was before (from the start of Avito until 2015 - 2016): we lived in a monolith, with monolithic bases and monolithic applications. At some point, these conditions began to prevent us from growing. On the one hand, we came up against the performance of the server with the main base, but this is not the main reason, since the performance issue can be solved, for example, using sharding. On the other hand, the monolith has a very complex logic, and at a certain stage of growth, the delivery of changes (releases) becomes very long and unpredictable: there are a lot of unobvious and complex dependencies (everything is closely related), testing is also time consuming, in general there are a lot of problems. The solution is to switch to microservice architecture. At this stage, we have a question with business transactions strongly tied to ACID, provided by a monolithic base: there is no clarity on how to migrate this business logic. When working with Avito, there are many different scenarios implemented by several services where integrity and consistency of data is very important, for example, buying a premium subscription, writing off money, applying services to a user, purchasing VAS packages — in unexpected circumstances or accidents, things might not go unexpectedly. according to plan. We found the solution in the sagas.

I like the technical description of the sagas that Kenneth Salem led in 1987 and Hector Garcia-Molina, one of the current members of the Oracle board of directors. As the problem was formulated: there are a relatively small number of long-lived transactions, which for a long time prevent the implementation of small, less demanding to resources and more frequent operations. As a desired result, you can give an example from life: surely many of you were queuing to otkerokopirovat documents, and the operator Xerox, if he had the task to copy a whole book, or just many copies, from time to time made copies of other members of the queue. But resource utilization is only part of the problem. The situation is aggravated by long locks when performing resource-intensive tasks, a cascade of which will line up in your DBMS. In addition, errors may occur during a long-running transaction: the transaction will not end and a rollback will begin. If the transaction was long, then the rollback will also take a long time, and probably will still be retry from the application. In general, "everything is quite interesting." The solution proposed in the technical description of "SAGAS": split the long transaction into parts.

It seems to me that many approached this without even reading this document. We have repeatedly talked about our defproc (deferred procedures implemented with pgq). For example, when blocking a user for fraud, we quickly execute a short transaction and respond to the client. In this short transaction, including, we set the task in the transaction queue, and then asynchronously, in small batches, for example, with ten announcements, block its declarations. We did this by implementing the transaction queues from Skype .

But our story today is a little different. We need to look at these problems from the other side: cutting the monolith into microservices built using the database per service pattern.

One of the most important parameters for us is reaching the maximum cutting speed. Therefore, we decided to transfer the old functionality and all the logic as it is to microservices, without changing anything at all. Additional requirements that we needed to fulfill:

• provide dependent data changes for business critical data;
• be able to set a strict order;
• Comply with 100% consistency - reconcile data even in case of accidents;
• guarantee the operation of transactions at all levels.

Under the above requirements, a solution in the form of an orchestrated saga is best suited.

Implementing orchestrated saga as a PG Saga service

This is the PG Saga service.

PG in the title, because as a service store uses synchronous PostgreSQL. What else is inside:

• API;
• executor;
• checker;
• healthchecker;
• compensator.

The diagram also shows the service owner of the sagas, and below - the services that will perform the steps of the saga. They may have different storages.

How it works

Consider the example of buying VAS-packages. VAS (Values-added services) - paid services to advertise.

First, the service owner of the saga must register the creation of the saga in the service saga

After that, it generates the saga class already with Payload.

Then, already in the saga service, executor picks up the previously created saga call from the storage and begins to perform it step by step. The first step in our case is the purchase of a premium subscription. At this moment, money is reserved in the billing service.

Then, in the user service, VAS operations are applied.

Then, VAS services are already operational, and packages of WAS are created. Further, other steps are possible, but they are not so important to us.

Accidents

Accidents can happen in any service, but there are well-known tricks on how to prepare for them. In a distributed system, it is important to know about these techniques. For example, one of the most important limitations is that the network is not always reliable. Approaches to solve interoperability problems in distributed systems:

1. Retry.
2. Mark each operation with an idempotent key. This is necessary to avoid duplication of operations. You can read more about idempotent keys in this article.
3. Compensate for transactions - an action characteristic of the sagas.

Transaction Compensation: How It Works

For each positive transaction, we need to describe the reverse actions: the business scenario of a step in case something goes wrong.

In our implementation, we offer the following compensation scenario:

If a step of the saga ended in failure, and we did a lot of retry, then there is a chance that the last repetition of the operation was a success, but we just did not get an answer. We will try to compensate for the transaction, although this step is not necessary if the service provider of the problem step is really broken and completely inaccessible.

In our example, it will look like this:

1. Turn off VAS packages.

1. We cancel user operation.

1. Cancellation of funds.

What to do if compensation does not work

Obviously, it is necessary to act in approximately the same scenario. Again, use retry, idempotent keys for compensating transactions, but if nothing works and this time, for example, the service is not available, you need to contact the service-owner of the saga, informing you that the saga has failed. Further more serious actions: to escalate the problem, for example, for manual investigation or launching automation to solve such problems.

What else is important: Imagine that any step of the saga service is unavailable. Surely, the initiator of these actions will still do some kind of retry. As a result, your saga service takes the first step, the second step, and its performer is unavailable, you cancel the second step, cancel the first step, and anomalies related to the lack of isolation may occur. In general, the saga service in this situation is engaged in useless work, which still generates pressure and errors.

How to do? Healthchecker should poll the services that perform the sag steps and see if they work. If the service has become unavailable, then there are two ways: the sagas that are in operation - to compensate, and the new sagas - either to prevent the creation of new instances (calls), or to create, without taking them into work with the executer, so that the service does not work superfluous actions.

Another accident scenario

Imagine that we are again making the same premium subscription.

1. We buy VAS-packages and reserve money.

1. We apply to the user of the service.

1. We create VAS packages.

It seems to be good. But suddenly, when the transaction was completed, it turns out that asynchronous replication is used in the user service and an accident has occurred on the master base. There can be several reasons for a replica lagging: the presence of a specific load on a replica, which either slows replication playback speed or blocks replication playback. In addition, the source (master) is overloaded, and a lag of sending changes appears on the source side. In general, for some reason, the replica was lagging behind, and the changes of the successfully completed step after the accident suddenly disappeared (result / condition).

To do this, we implement another component in the system - use the checker. The checker goes through all the steps of successful sagas through time that is obviously greater than all possible lags (for example, after 12 hours), and checks whether they are still successfully completed. If the step is suddenly not executed, the saga rolls back.

There may still be situations when, after 12 hours, there is already nothing left to cancel - everything changes and moves. In this case, instead of the cancellation scenario, the solution might be to signal the service of the owner of the saga that this operation was not completed. If the cancellation operation is impossible, say, you need to cancel after the money is credited to the user, and its balance is already zero, and the money cannot be written off. We have such scenarios are always solved in the direction of the user. You may have another principle, this is consistent with the representatives of the product.

As a result, as you can see, in different places for the integration with the sagas service you need to implement a lot of different logic. Therefore, when client teams want to create a saga, they will have a very large set of very unobvious tasks. First of all, we create a saga so that duplication does not work out, for this we work with some idempotent operation of creating a saga and its tracking. Also, the services need to realize the ability to track every step of each saga, in order not to execute it twice on the one hand, and to be able to answer on the other hand whether it was actually executed. And all these mechanisms must somehow be maintained so that the service storages do not overflow. In addition, there are many languages ​​in which services can be written, and a huge selection of repositories. At each stage, you need to understand the theory and implement all this logic on different parts. If you do not, you can make a whole bunch of mistakes.

There are many correct ways, but there are no less situations where you can “shoot yourself a limb”. In order for the sagas to work correctly, you need to encapsulate all the above mechanisms in client libraries that will transparently implement them for your clients.

An example of saga generation logic that can be hidden in the client library

You can do it differently, but I suggest the following approach.

1. We get the request ID, on which we have to create a saga.
2. We go to the saga service, we get its unique identifier, we save it in the local storage in conjunction with the request ID from item 1.
3. We start the saga with payload in the saga service. An important caveat: I suggest local operations of the service, which creates a saga, to arrange as the first step of the saga.
4. There is a kind of race when the saga service can perform this step (point 3), and our backend, which initiates the creation of the saga, will also perform it. To do this, we do idempotent operations everywhere: one of them performs it, and the second call simply receives “OK”.
5. We call the first step (point 4) and only after that we respond to the client who initiated this action.

In this example, we are working with the saga as a database. You can send a request, and then the connection may be terminated, but the action will be executed. Here is about the same approach.

How to check it all

It is necessary to cover the entire service sagas tests. Most likely, you will make changes, and tests written at the start will help to avoid unexpected surprises. In addition, it is necessary to check the sagas themselves. For example, how we have arranged the testing of the service of the sagas and the testing of the sequence of the sagas in the framework of one transaction. There are different blocks of tests. If we are talking about the saga service, he is able to perform positive transactions and compensation transactions, if the compensation does not work, he informs the service owner of the sagas. We write tests in general, to work with an abstract saga.

On the other hand, positive transactions and compensation transactions on services that perform the steps of the sagas are the same simple API, and tests of this part are in the area of ​​responsibility of the team owning this service.

And then the team owner of the saga writes end-to-end tests, where it checks that all business logic works correctly when the saga is executed. The end-to-end test takes place on a full-fledged dev-environment, all instances of services, including the service sagas, are raised, and the business scenario is being tested there.

Total:

• write more unit tests;
• write integration tests;
• write end-to-end tests.
  
  The next step is the CDC. Microservice architecture affects the specifics of tests. In Avito, we adopted the following approach to testing microservice architecture: Consumer-Driven Contracts. This approach helps, above all, to highlight the problems that can be identified in the end-to-end tests, but the end-to-end test is “very expensive.”

What is the essence of the CDC? There is a service that provides a contract. He has an API - this is a provider. And there is another service that calls the API, that is, uses the contract - the consumer.

Service-consumer writes tests for the provider contract, and the tests that only the contract will check are non-functional tests. It is important for us to ensure that if we change the API, we will not break steps in this context. After we have written the tests, another service broker element appears - it records the information on CDC tests. Each time the provider’s service changes, it will raise isolated environments and run tests that consumer has written. What is the result: the team that generates the sagas, writes tests for all the steps of the saga and registers them.

Frol Kryuchkov told RIT ++ about how the Avito implemented the CDC approach for testing microservices. Abstracts can be found on the website Backend.conf - I recommend to read.

Types of sagas

In order of function call

a) unordered - the functions of the saga are called in any order and do not wait for each other to complete;
b) ordered - the functions of the saga are called in the specified order, one after another, the next is not called until the previous one is completed;
c) mixed - for some of the functions, an order is set, but for some, it is not, but it is set before or after what stages they are to be performed.

Consider a specific scenario. In the same scenario of buying a premium subscription, the first step is to reserve money. Now we can make changes to the user and creating bonus packages in parallel, and we will send notifications to the user only when these two steps are completed.

Upon receipt of the result of a function call

a) synchronous - the result of the function is known immediately;
b) asynchronous - the function returns immediately “OK”, and the result is returned later, via the callback API of the sagas service from the client service.

I want to warn you against an error: it is better not to take the synchronous steps of the sagas, especially when implementing an orchestrated saga. If you take simultaneous steps of the sagas, the saga service will wait until this step is completed. This is an extra burden, unnecessary problems in the service of the sagas, since it is one, and there are many participants in the sagas.

Scaling sagas

Scaling depends on the size of the system you are planning. Consider a single instance of the repository:

• one handler of steps of the saga, we process steps with batches;
• n handlers, we implement the "comb" - we take the steps of the remainder of the division: when each executor gets its steps.
• n handlers and skip locked - will be even more efficient and more flexible.

And only then, if you know in advance that you will run into the performance of a single server in a DBMS, you need to do a sharding - n instances of databases that will work with your data set. Sharding can be hidden behind the API service sagas.

More flexibility

In addition, in this pattern, at least in theory, the client service (performing the saga step) can access and fit into the saga service, and participation in the saga can also be optional. There may be another scenario: if you have already sent an email, you cannot compensate for the action - you cannot return the letter back. But you can send a new letter, that the previous one was wrong, and it looks like so-so. It is better to use the scenario when the saga will be played only forward, without any compensation. If it does not play forward, then it is necessary to inform the service owner of the saga about the problem.

: , . — . : .

, , . , . - , . . defproc , , .

How to do it? , , , , , - , . . . : , , .

-, , . , , . , , . . — , , .

ACID —

, , . . — durability. . . , . - , - - ,

$$$Ti reads => other rtransaction writes => Tj (or Ci) writes$

— - , - , , - , . , - , - .

$$$Ti writes => other transaction reads => Ci writes$

— .

$$$Ti reads =>other transaction writes =>Tj reads$

:

1. , , , , .
2. , . , , , , , .
3. .
4. payload . eventual consistency — , , , . , , , -.

Monitoring

. , . . checker. . , .

(50%, 75%, 95%, 99%), , - .

, — , . . , - . , — .

. , - ( ) . healthchecker endpoint' info (keep-alive) .

. -. -, - , - . , , , end-to-end. - . , , — .
. .

:

, healthchecker, - , . , . .

, . , , . . choreography — - . , choreography- , . choreography , . , . , , , .

. , , . , + .

API

, - - ( API ), , API. API . — . API , , 100% .

, , , , . — , , . .

, , , . ( ) .

, , , , .

. , , .

saga call ID

. API , .

—

- legacy . , ( «» ). « »? - , , , , - , . , , , . , « », , -. . — . , .

, , . , , , , . , , . .

, . .