Warehouse management system using CQRS and Event Sourcing. Design

So, after making the requirements described in Part 1, you can proceed to the design of the system.

Our main task in design, as is clear from the title of the article, is to achieve separation of interfaces into Query and Command, in order to subsequently divide business scenarios into those that will read data (Query interfaces) and those that will modify data (Command interfaces). And also to provide the minimum time of expectation (latency) on the update of the data accessible through Query, after we changed the data through Command.

Why CQRS?

Recently, the separation of interface methods ( CQS ), and subsequently the interfaces themselves on Query / Command, has become a popular trend in the development of application architecture. But Magento CQRS is used not because of its popularity, but because for us it is sometimes the only way to build a flexible, extensible (adapting to specific needs) solution with the ability to independently scale Read and Write operations.

In fact, we came to CQRS as a result of the comprehension and widespread use of SOLID in writing code. CQRS is the implementation of The Single Responsibility Principle and the Interface Segregation Principle and a departure from the classic CRUD.
')
The elements of CQRS appeared quite a long time ago when we asked ourselves how to scale the EAV data model (entity-attribute-value) for read operations and introduced index aggregation tables for this.

In more detail about CQRS, and what it means for us, you can listen to the presentation that I did at MageCONF 2016 ( video , slides ).

Domain Entity Description

In the Inventory domain, we have six main domain models:

• Source - the entity responsible for the presentation of any physical warehouse where there may be a product (store, warehouse).
  
  
• Source Item - an entity-bundle, represents the quantity of a specific product ( SKU ) on a specific physical storage. And also keeps the status of whether the product is available for sale at the moment.
  
  
• Stock - virtual entity responsible for the presentation of stocks in several physical warehouses (Source) at the same time. It is an aggregation of physical warehouses. The connection between warehouses and Stock (which warehouses are included in the aggregation) is set by the administrator in the admin panel or via an API call.
  
  
• Stock Item is an index entity based on the relationship between Source and Stock, and is the quantity of a specific product ( SKU ) available on a specific Stock, i.e. virtual aggregation.
  
  
• Sales Channel - a sales channel through which sales are made to the end customer In the case of Magento, this may be (Website, Store, Store View), but the sales channel may be determined by the seller on its own, so for some sellers it may be a Country (Country), or a large Wholesale customer (Wholesale) may act as an independent sales channel. In fact, the sales channel is the context within which the sale takes place, which helps us to clearly define the Stock to be used during the execution of a business operation.
  
  
• Reservation is a reservation object that is created in order to have the current level of goods that we have for sale between order creation events and a decrease in the number of goods in specific physical warehouses. In fact, the reservation object helps us to get rid of the need to perform checks, locks and deduct goods from the inventory of physical warehouses (Source Item) during the order placement operation, while the value of goods available for sale within a certain Stock changes immediately.

This diagram represents the interaction of the entities described above:



Thus, we get the separation of interfaces into Query and Command:

Theory of operation

One of our main tasks is to unload the order placement process as much as possible. And ideally make this operation one that will not change the state of the system (State Modification). This will eliminate unnecessary locks and improve the scaling of the checkout.

The diagrams above show that such operations as rendering a category page will be performed using only the Query API (StockItem), where we need to display the number of products that we can sell in a specific context (SalesChannel).

Synchronization with an external ERP or PIM system will use the Command API (SourceItem) and update the quantity of goods in specific warehouses, after which the re-indexing, which will re-compile StockItem, will allow these updates to be visible on the front.

In the case of an order placement operation, everything becomes more interesting.

Placing an Order in Steps

The operation of placing an order is divided into two operations: directly placing an order , in which the buyer participates and which ends with payment and confirmation of acceptance of the order for execution; and order processing , which occurs after the fact (with a certain delay of several milliseconds to minutes and hours) whose main task is to determine from which warehouse ( or warehouses ) we have to deliver to our customer and calibrate the quantity of goods in these warehouses after the order has been completed.

Actually, consider these operations on the example in more detail.
Suppose we have 3 physical warehouses: Source A, Source B, Source C. Where SKU-1 goods are stored in such quantity:

• SourceItem A - 20
• SourceItem B - 25
• SourceItem C - 10

We have only one sales channel (Sales Channel), the role of which is performed by Website.
For this sales channel, we have created a virtual aggregation of Stock A, with which all current physical warehouses are connected (Source A, Source B, Source C). Get StockItem A for SKU-1 has a quantity of 20 + 25 + 10 = 55

Accordingly, when the buyer enters the Website, the system accurately determines the Stock, which must be applied to determine the quantity of goods, and uses Stock Items within this flow for all products (SKU) in the category, in our case SKU-1.

Placing an order

Let the buyer decide to buy 30 units of SKU-1 product.

1. We decide whether we can complete the sale (do we have enough goods for sale), the number of StockItem A for SKU-1 = 55 minus the number of all reservations for product SKU-1 at Stock A, in our case 0 (so far no reservation has been created), 55 - 0> 30, then we decide that we can sell 30 units of SKU-1 goods.
   
   
2. At the time of placing an order, we are agnostic to the fact from which physical warehouses as a result will be written off, so we do not work with Source Item entities. * This topic will be discussed in a separate article, which will be fully devoted to the algorithm for selecting warehouses for delivery as part of order fulfillment.
   
   
3. At the same time, we cannot reduce the number of SKU-1 on StockItem, since this is a Read projection created only for reading. Therefore, we create a reservation (Reservation) for SKU-1 on Stock A in the amount of 30 units. Creating a reservation is an Append Only operation that is added without any checks.
   
   
4. The order team itself can be queued for further processing.

The current state of the system:
Quantity of goods SKU-1 in warehouses:

• SourceItem A - 20
• SourceItem B - 25
• SourceItem C - 10

The quantity of SKU-1 for StockItem A is 55 (unchanged).
Reservation for SKU-1 on Stock A in the amount of (-30).

While we did not have time to process this order, for example, because of the high latency, another buyer comes to our site, who also wants to purchase SKU-1 goods in the amount of 10 units.

The system begins to perform the same steps as described above.
Check whether we can sell 10 units of SKU-1. The check is carried out in the following way: the number of StockItem A for SKU-1 = 55 minus the number of all reservations for product SKU-1 on Stock A, in our case (-30), 55 - 30 = 25> 10, we decide to sell 10 units of goods SKU-1 can.

In fact, a state in Event Sourcing is defined as a projection of aggregated data (in our case StockItem), with the addition of all the events that were received for the time delta since the formation of this projection (in our case it is Reservation).

Order Processing

At this stage, we must determine which physical warehouses will take part in the delivery, and for these warehouses, reduce the value of the quantity of goods shipped.
For this part, the responsible algorithm that will decide on the choice of warehouses (will be described in the next article). For example, the algorithm decided that it would be cheaper for the seller to send 30 goods from the Source C and Source B warehouses.

Accordingly, the quantity of SKU-1 goods in warehouses should change:

• SourceItem A: 20
• SourceItem B: 25 - 20 = 5
• SourceItem C: 10 - 10 = 0

After that we create another Reservation object on SKU-1 within Stock A in the amount of (+30) units in order to “zero out” the last created reserve object (-30 + 30 = 0). After that, a team should be created to update the StockItem index (which is built as an aggregation of the presence of goods in physical warehouses), which can also be asynchronous.

Magento MSI (Multi Source Inventory)

This article is the second article in the “Warehouse Management System Using CQRS and Event Sourcing” cycle within which the collection of requirements, design and development of a warehouse management system will be considered using Magento 2 as an example.

An open project, where development is underway, and where engineers from the community are involved, as well as where you can get acquainted with the current state of the project and documentation, is available here .