Smart Integration Plugs
Gone are the days when banking systems were large monolithic applications that were slowly updated to meet the requirements of the regulator. Now there are hundreds of interconnecting subsystems and thousands of modules that are continuously evolving with new business requirements and rapidly changing IT space. Hundreds of individual combat units — Agile teams — have been thrown at the development of all this.
Imagine it? Yes, it captures the spirit. Nevertheless, this is the usual everyday life of the developers of the Unified Frontal System of Sberbank. Two hundred teams are simultaneously engaged in the development of the platform, daily solving complex and non-trivial tasks aimed at optimizing and synchronizing processes.
You can talk about synchronization and optimization for a long time, this topic deserves a separate book. In this article we will cover only a small part - the optimization of the development of integration interaction, we will share our experience and tell you how to integrate with a hundred systems and not wait for anyone.
')
The question that worries us in the first place is how all teams, each of which depends on many other teams, have time to make their development with high quality, in compliance with all deadlines, and so that when connected to adjacent systems on the integration ground you get the expected result?
And we found two answers to it: a single information space and maximum automation of development and deployment.
Now our integration center is the “Unified Register of Integration Interaction”. The registry contains information on all subsystems and services, details of integration interaction, attribute composition, description of data sources and use register, formats, protocols and other characteristics necessary for an accurate and unambiguous description of each interaction. In addition, it is associated with similar registries of other bank systems. All this forms a single information space.
With a single information space, it is clear what about automation? The more code is generated automatically, the lower the cost and the higher the quality of the final product. For the development of integration interaction, this is doubly important, since many of the separately developed systems must cooperate and understand each other together.
As you probably guessed, the integration interaction register is needed not only for reference information. Based on the registry data, the automatic generation of integration components (integration layer) takes place. This saves teams from additional routine development, forms a unified integration architecture and ensures consistency of interaction in accordance with the stated contracts.
Yes, really comfortable, but that's not all. The team got all its integration into the registry, received the generated integration layer and ... stop. The functionality of the modules developed by the team depends on the adjacent systems that are themselves in the process of development. How to be? In addition, at the end of the sprint, the Product Owner will want to see the result of the team’s work. Perhaps, along with the stakeholders, he would like to make some backlog adjustments. Agile, after all.
That's right, you need to silence the missing integration interactions. And then all the teams rushed to write their stubs.
There is a catastrophic lack of time, there is a great temptation to stifle the interaction, spending minimal effort. For example, add the return result directly to the code of the calling function. Not quite honestly, we agree, but you don’t have to waste time writing separate components, deploying and setting up various message brokers. Yes, the next sprint will have to change a lot, yes, there is no guarantee that adjacent systems will work similarly to this implementation, but there is no time - you need to complete the sprint, it’s not up to cost optimization.
And here again the Unified Register of Integration Interaction comes to the rescue. If it is already used to generate integration components, then what prevents you from immediately generating the corresponding stubs, automatically deploying them to the common infrastructure accessible from the development and demonstration environment. And if we add to this a separate tool for creating test scenarios, so that not only developers could write them, but also testers or analysts who work out all the interactions in detail and are well aware of the formats, attributes, and use cases of each service. And if we additionally combine test scripts written by different teams into a common directory, then we will get real “smart stubs”.
Our smart plugs now support the JMS messaging standard. This allows you to emulate most of the interactions of the Unified frontal system with adjacent systems. At the same time, support for four types of interaction is implemented:
To select a response template, the conditions for incoming messages using XPath notation are used. The generation of responses from specified templates occurs with the help of FreeMarker. It supports the validation of the request / response on the XSD scheme. All requests and responses are logged to the database.
We mentioned above that the main developers of stubs and test scripts are testers and analysts. Analysts are directly involved in the elaboration of integration interactions, describe the attribute composition of services, use cases and sequence of calls. Testers form test scripts based on user stories (User Story). With this in mind, all settings for stubs and writing test scripts are done via a user-friendly web interface, with no programming skills required. Much attention is paid to working with message logs: filtering, viewing the list of calls and detailed information on each call. This functionality is necessary when analyzing the work of stubs and for understanding the operation of test scripts.
Now imagine that some of the related systems are ready, some are released in the current sprint, and something will be developed later. The centralized release of the integration layer allows you to manage the configuration of adapters and plugs from a single Parameter Management System (EMS). The adapters are configured for operation with real systems or plugs through the EMS, while switching occurs in real time without restarting and even more so without reassembling applications.
Taking into account the fact that smart plugs are provided according to the service model, and teams do not need to deploy their own infrastructure (search for servers, install and configure something), new plugs can be created and started using in a few clicks. Agree, in terms of tight deadlines, this is a big help.
Simplifying the development of stubs is, of course, good. But is it possible to completely do without any costs? Oddly enough, but yes. The more plugs are developed, the higher the likelihood of their reuse. To simplify the search for implemented stubs, we have a separate registry. Before we develop our own stub, we search the registry for the master system and the service for all the test scripts already written for this interaction. And if the necessary plugs are already developed, then we simply use them at no cost. If not, then either we are refining something in the current scenarios, or we form new stubs. All separate areas of testing can be used, it is very convenient, because it allows you to avoid mutual influence:
The introduction of a common information space for integration of the ESF, together with the automation of the development and deployment of the integration layer and smart plugs, has significantly reduced costs, improved the quality and consistency of the integration interaction, and also provided the ability to synchronously develop many commands using Agile methodology.
Much has already been done, but judging by the requests of the teams and by the formed roadmap, this is only the beginning. In the near future:
There are many tasks and everything needs to be done yesterday. Therefore, we encourage joint development. Each team can modify the functionality of smart stubs or registry. Having made the necessary refinement for themselves, the team implements this for everyone else. Joint development of tools allows you to significantly accelerate the development of products and the timing of the implementation of the roadmap.
And what do you think about this? If you are close and interested in the topic of integration, we will be happy to talk in the comments to the post.
Imagine it? Yes, it captures the spirit. Nevertheless, this is the usual everyday life of the developers of the Unified Frontal System of Sberbank. Two hundred teams are simultaneously engaged in the development of the platform, daily solving complex and non-trivial tasks aimed at optimizing and synchronizing processes.
You can talk about synchronization and optimization for a long time, this topic deserves a separate book. In this article we will cover only a small part - the optimization of the development of integration interaction, we will share our experience and tell you how to integrate with a hundred systems and not wait for anyone.
')
The question that worries us in the first place is how all teams, each of which depends on many other teams, have time to make their development with high quality, in compliance with all deadlines, and so that when connected to adjacent systems on the integration ground you get the expected result?
And we found two answers to it: a single information space and maximum automation of development and deployment.
Now our integration center is the “Unified Register of Integration Interaction”. The registry contains information on all subsystems and services, details of integration interaction, attribute composition, description of data sources and use register, formats, protocols and other characteristics necessary for an accurate and unambiguous description of each interaction. In addition, it is associated with similar registries of other bank systems. All this forms a single information space.
With a single information space, it is clear what about automation? The more code is generated automatically, the lower the cost and the higher the quality of the final product. For the development of integration interaction, this is doubly important, since many of the separately developed systems must cooperate and understand each other together.
As you probably guessed, the integration interaction register is needed not only for reference information. Based on the registry data, the automatic generation of integration components (integration layer) takes place. This saves teams from additional routine development, forms a unified integration architecture and ensures consistency of interaction in accordance with the stated contracts.
Yes, really comfortable, but that's not all. The team got all its integration into the registry, received the generated integration layer and ... stop. The functionality of the modules developed by the team depends on the adjacent systems that are themselves in the process of development. How to be? In addition, at the end of the sprint, the Product Owner will want to see the result of the team’s work. Perhaps, along with the stakeholders, he would like to make some backlog adjustments. Agile, after all.
What to do?
That's right, you need to silence the missing integration interactions. And then all the teams rushed to write their stubs.
There is a catastrophic lack of time, there is a great temptation to stifle the interaction, spending minimal effort. For example, add the return result directly to the code of the calling function. Not quite honestly, we agree, but you don’t have to waste time writing separate components, deploying and setting up various message brokers. Yes, the next sprint will have to change a lot, yes, there is no guarantee that adjacent systems will work similarly to this implementation, but there is no time - you need to complete the sprint, it’s not up to cost optimization.
And here again the Unified Register of Integration Interaction comes to the rescue. If it is already used to generate integration components, then what prevents you from immediately generating the corresponding stubs, automatically deploying them to the common infrastructure accessible from the development and demonstration environment. And if we add to this a separate tool for creating test scenarios, so that not only developers could write them, but also testers or analysts who work out all the interactions in detail and are well aware of the formats, attributes, and use cases of each service. And if we additionally combine test scripts written by different teams into a common directory, then we will get real “smart stubs”.
EFS smart plugs
Our smart plugs now support the JMS messaging standard. This allows you to emulate most of the interactions of the Unified frontal system with adjacent systems. At the same time, support for four types of interaction is implemented:
- ESF request / external response;
- ESF request / no external speaker response (notification);
- request from the external speaker / no response from the EFS (notification);
- request from the external speaker / with the answer from the ESF.
To select a response template, the conditions for incoming messages using XPath notation are used. The generation of responses from specified templates occurs with the help of FreeMarker. It supports the validation of the request / response on the XSD scheme. All requests and responses are logged to the database.
We mentioned above that the main developers of stubs and test scripts are testers and analysts. Analysts are directly involved in the elaboration of integration interactions, describe the attribute composition of services, use cases and sequence of calls. Testers form test scripts based on user stories (User Story). With this in mind, all settings for stubs and writing test scripts are done via a user-friendly web interface, with no programming skills required. Much attention is paid to working with message logs: filtering, viewing the list of calls and detailed information on each call. This functionality is necessary when analyzing the work of stubs and for understanding the operation of test scripts.
Now imagine that some of the related systems are ready, some are released in the current sprint, and something will be developed later. The centralized release of the integration layer allows you to manage the configuration of adapters and plugs from a single Parameter Management System (EMS). The adapters are configured for operation with real systems or plugs through the EMS, while switching occurs in real time without restarting and even more so without reassembling applications.
Taking into account the fact that smart plugs are provided according to the service model, and teams do not need to deploy their own infrastructure (search for servers, install and configure something), new plugs can be created and started using in a few clicks. Agree, in terms of tight deadlines, this is a big help.
Simplifying the development of stubs is, of course, good. But is it possible to completely do without any costs? Oddly enough, but yes. The more plugs are developed, the higher the likelihood of their reuse. To simplify the search for implemented stubs, we have a separate registry. Before we develop our own stub, we search the registry for the master system and the service for all the test scripts already written for this interaction. And if the necessary plugs are already developed, then we simply use them at no cost. If not, then either we are refining something in the current scenarios, or we form new stubs. All separate areas of testing can be used, it is very convenient, because it allows you to avoid mutual influence:
And finally
The introduction of a common information space for integration of the ESF, together with the automation of the development and deployment of the integration layer and smart plugs, has significantly reduced costs, improved the quality and consistency of the integration interaction, and also provided the ability to synchronously develop many commands using Agile methodology.
Much has already been done, but judging by the requests of the teams and by the formed roadmap, this is only the beginning. In the near future:
- Expansion of supported protocols and formats: http, kafka, json.
- Support for complex test scenarios that include multiple calls, and save intermediate results in the context of a single test scenario.
- The ability to call external services and record real responses for later use in the stubs.
- Plugin support for enhanced emulation capabilities.
There are many tasks and everything needs to be done yesterday. Therefore, we encourage joint development. Each team can modify the functionality of smart stubs or registry. Having made the necessary refinement for themselves, the team implements this for everyone else. Joint development of tools allows you to significantly accelerate the development of products and the timing of the implementation of the roadmap.
And what do you think about this? If you are close and interested in the topic of integration, we will be happy to talk in the comments to the post.
Source: https://habr.com/ru/post/330376/
All Articles