How and why to move from service-oriented architecture to microservices
Hello, my name is Alexey, I am the chief IT architect of Renaissance Credit Bank. Ten years ago, we, like many companies, accelerated their development thanks to a service-oriented architecture (SOA). But over time, the requirements for architecture changed, and serious questions began to arise about this paradigm. In the end, we decided to move from the ESB integration bus to microservices. In our example, I will tell you why you should think about the effectiveness of SOA and what you can do if this model does not suit you either.
It all started with the fact that we appreciated the structure of our integration bus. The diagram shows a typical integration flow situation for an ESB.
')
There may be several dozens of such blocks (up to a hundred!), And they are closely intertwined. If you need to make a new product and make some changes, then there will not be enough resources to process this hefty monolith. And this is just a general view. We studied SOA from all sides and identified seven major issues:
The traditional service bus, focused on waterfall, with its complex and slow processes does not fit into the market conditions. Today, customers are aware of flexible development methodologies. They may not know the differences between agile and waterfall, but they want to get the first results - from the appearance of the idea to the prototype in operation - not in 6-8, but in 2-3 months, but better in general after one. Let it be the MVP, but it is important for the customer to check the business idea as soon as possible, to understand that the decision will “take off”.
In such conditions, traditional, horizontally oriented teams are inferior to vertically oriented teams that work with all the components of the product - from the interaction channel to the back end. The question arises: what to do with the integration bus, which has turned into a monolith because of the reuse of services? We need a new technological approach.
We began to develop an alternative to SOA with the wording of the basic requirements:
In accordance with the requirements, we have formed a pyramid of technologies:
Now about the components of each level separately:
1. Methodology level
2. The level of infrastructure
3. Applied architecture
Please note: there is no mandatory reuse requirement for MSA!
4. The level of the data model
5. Level of integration.
Based on the new combination of technologies, we presented what we want to come to. The figure below is a diagram of our current infrastructure with an integration bus. Next - the scheme to which we aspire.
What is changing? Initially, at the back office level, any ABS has business logic and a data source. We strive to ensure that back-office systems are reduced exclusively to storage and "nuclear" functionality, where changes would be rare. And the product logic went to the level of microservices, allowing flexible change and create products, divided by domain.
On the scale of the channel, we are planning on the basis of microservices to form and manage the logic “spread” over the front end layer. As a result, the front-end applications themselves will contain only channel logic, that is, native applications for automating the necessary query service channels.
All access control will be implemented through the API Management portal, where all APIs will be described and published. Here, any developer will be able to get all the information about them and join the development factory. It uses a continuous cycle of work using GitLab technologies - planning, development, testing, production and operation.
API Management and Open API
Changes of this scale do not pass without difficulties. They are mainly related to the breaking up of monolithic box systems into microservices, as well as the restoration of connections in the “black boxes” of our ESB and Data Silo. With the model of microservices, the use of the BPM engine for building business processes is no longer relevant. And now the question of its alternative becomes very important for us - event-driven-architecture and choreography. We also plan to move from pure DevOps to DevSecOps, which will include ITSec requirements, and divide our Data Silo into domains. Accomplishing these tasks will require actively collecting experience to test and maximize the use of the concepts described.
New ESB Problems
It all started with the fact that we appreciated the structure of our integration bus. The diagram shows a typical integration flow situation for an ESB.
')
There may be several dozens of such blocks (up to a hundred!), And they are closely intertwined. If you need to make a new product and make some changes, then there will not be enough resources to process this hefty monolith. And this is just a general view. We studied SOA from all sides and identified seven major issues:
- Focus on service reuse. This rule was the cornerstone of SOA, but in the end it led to the fact that all the teams (front, back and integration) were so tightly coupled that any update to the integration required a regression test of almost the entire bus!
- Heavy technological stack. Initially, a single ESB container was an advantage, and this allowed us to quickly deploy services. But today, a limited stack of technologies, libraries, and programming languages ​​is already noticeably hindering development.
- Turning the ESB into a bottleneck for all processes. Over time, the tire acquires functions and turns into a full-fledged bank information system, with a waterfall approach and “cutting” integration flows. The load of the ESB team is greatly increasing, which hinders the work of other development teams.
- DataSilo. In SOA, service interfaces are separate from implementation. But how separated are the data used by the services? One service accesses the data of another, the dblink mechanism is used, confusion arises and a “data silo”.
- Mixing integration patterns. Strictly speaking, there are two of them for SOA: the classic MessageBroker, where messages go between systems for exchanging chunks of data, and the hub of services, where we host some services that “have nowhere else to stick”. When mixing these templates, the ESB turns into an absolutely “black box”.
- The formation of "shadow IT". Business customers often form their own IT-teams for the development of critical systems for themselves, but they cannot use the bus for integration. This is how the number of “left” connections to information systems grows.
- Lack of ITSec (information security) support at the concept level. A little explanation needs to be made here: firstly, the completely “Western” concept of the ESB does not take into account the peculiarities of Russian legislation (for example, the protection of personal data during transmission from the source to the receiver), and secondly, the ESB container is a perimeter that works effectively, when inside it is no longer required to comply with the special requirements of ITSec.
New customer needs
The traditional service bus, focused on waterfall, with its complex and slow processes does not fit into the market conditions. Today, customers are aware of flexible development methodologies. They may not know the differences between agile and waterfall, but they want to get the first results - from the appearance of the idea to the prototype in operation - not in 6-8, but in 2-3 months, but better in general after one. Let it be the MVP, but it is important for the customer to check the business idea as soon as possible, to understand that the decision will “take off”.
In such conditions, traditional, horizontally oriented teams are inferior to vertically oriented teams that work with all the components of the product - from the interaction channel to the back end. The question arises: what to do with the integration bus, which has turned into a monolith because of the reuse of services? We need a new technological approach.
The foundation of the new approach
We began to develop an alternative to SOA with the wording of the basic requirements:
- The possibility of rapid piloting of ideas. This will require a simple (regular, repetitive) structure of links between systems, rapid deployment and versioning.
- Support for agile development. Easy connection of new, vertical commands, access to the level of “development factory”, with automation of routine tasks.
- Presence of an ecosystem with development teams of different types : internal, external, partnership. Providing interfaces for external access, control of this access and billing.
Technology pyramid
In accordance with the requirements, we have formed a pyramid of technologies:
Now about the components of each level separately:
1. Methodology level
- Agile . The flexible approach now causes a lot of emotions and opposing opinions (on the topic of applicability in organizations of various sizes). For ourselves, we have formulated the most important thing: this approach is the basis for structuring requirements, rapid prototyping and creating MVP - testing ideas of products.
- DevOps. This paradigm requires maximum automation and “blurring the boundaries” between the development and maintenance of systems. It allows you to avoid wasting time on routine deployment and tracking operations.
- Factory development . The movement of artifacts from the analytics stage to the deployment and operation of the created product must be continuous, without re-creating artifacts at each next stage. For example, there should not be a situation when the service interface is first described in some format by an analyst, and then the developer manually creates the same interface anew, but in a different tool.
2. The level of infrastructure
- Containerization (Docker) . The only option to provide microservices at this level is to leave a single bus container and use separate containers for each service instance. This means that the use of "heavy" application servers that run a wide range of libraries at startup, is also not suitable. The container should be as lightweight and configurable as possible in order to ensure that only the set of functionality necessary for the service is launched. And from this point of view, Docker is a great fit.
- Container orchestrator . Containers should be managed by a tool that solves typical problems of fault tolerance, balancing and unification of deployment / shutdown / launch. At the same time, this tool cannot be an analogue of a tire with its monolithic container.
3. Applied architecture
- Microservice architecture (MSA) . The question of the exact definition of microservice is still open. But for the full development of integration architectures and systems, we identified the following key properties of microservices:
| Property | Explanation | |
| one | Interface separation and service implementation | This property is inherited from SOA and requires that the implementation change does not require an interface change. And the service call required only interface knowledge, without understanding the subtleties of the service implementation. |
| 2 | Good granularity | Microservices should be relatively small (“the rule of two pizzas”) and separated from each other so that changes in functionality in any subject area are concentrated as much as possible in one microservice. |
| 3 | Separation of services at all levels | Microservices must be completely separated from each other. At the interface level and at the execution level - each has its own execution container. At the data level, microservice has access only to "its" data and knows nothing about the features of the database of the neighboring microservice. |
| four | Unified interaction | If microservice requires data that is accessed by another microservice, the interface call should work. In no case do not access through the database in the neighboring scheme. |
Please note: there is no mandatory reuse requirement for MSA!
4. The level of the data model
- DDD (Domain Driven Design) . One of the most difficult issues is the set of rules by which microservices are created, and ensuring their connection with the earlier stages of analytic software development. We tried to push off from the concept of DDD with two main objectives. First, it allows you to create domains in the domain and successfully link them with product teams (agile!). Secondly, it helps to form microservice as an API for working with a specific business object - corresponding to a resource for a RESTful service.
- Domains correspond to the products of the bank. This allows you to move away from the classical division to the front, back and integration, to come to unity with the product teams.
5. Level of integration.
- API Management and API First Approach . Integration with the use of the bus means “cutting” integration flows from the front to the back with the reuse of services. In the new version, we focus on the API First approach. Back-to-back systems prepare the most common APIs. The integration is based on the principle of the base crystal: the developers of the frontal systems select the API calls they need, published on the API Management portal.
- Open API . Open API implies the use of API Management system for organizing access and operation of fundamentally different categories of developers - external, internal, partner ecosystem participants. In fact, we get the public API of the organization.
Architecture changes
Based on the new combination of technologies, we presented what we want to come to. The figure below is a diagram of our current infrastructure with an integration bus. Next - the scheme to which we aspire.
What is changing? Initially, at the back office level, any ABS has business logic and a data source. We strive to ensure that back-office systems are reduced exclusively to storage and "nuclear" functionality, where changes would be rare. And the product logic went to the level of microservices, allowing flexible change and create products, divided by domain.
On the scale of the channel, we are planning on the basis of microservices to form and manage the logic “spread” over the front end layer. As a result, the front-end applications themselves will contain only channel logic, that is, native applications for automating the necessary query service channels.
All access control will be implemented through the API Management portal, where all APIs will be described and published. Here, any developer will be able to get all the information about them and join the development factory. It uses a continuous cycle of work using GitLab technologies - planning, development, testing, production and operation.
API Management and Open API
What's next?
Changes of this scale do not pass without difficulties. They are mainly related to the breaking up of monolithic box systems into microservices, as well as the restoration of connections in the “black boxes” of our ESB and Data Silo. With the model of microservices, the use of the BPM engine for building business processes is no longer relevant. And now the question of its alternative becomes very important for us - event-driven-architecture and choreography. We also plan to move from pure DevOps to DevSecOps, which will include ITSec requirements, and divide our Data Silo into domains. Accomplishing these tasks will require actively collecting experience to test and maximize the use of the concepts described.
Source: https://habr.com/ru/post/351804/
All Articles