Moving web portal: from monolith to microservice architecture
We share the experience of modernizing the architecture of a web portal, which made it possible to make the product convenient, stable and fault-tolerant: attendance increased from 350,000 to 7 million users! The story will be useful to those who are planning to expand their product. To begin with, let's look at what factors may hinder the work and development of the web portal:
Case
We were approached by a client whose web portal needed further development. At a certain point, a ceiling of 350,000 unique visitors was reached monthly. This seemed to be insufficient - the project clearly had prospects, and we had to figure out exactly what was in the way of further scaling the resource.
')
Audit
It turned out that the old Bitrix engine was not adapted to the loads and scaling. Every one of the components of the system closely intersected and depended on each other. Constantly eliminating errors was such a painful exercise that we proposed a flexible and understandable microservice architecture. In them, each component is sufficiently independent that the errors inside it do not “touch” the other elements of the system.
The basis of the architecture remained Bitrix - at least, authorization is tied to it, but where it was possible and promised greater efficiency, we transferred the functionality to microservices. They became the cornerstone of the project.
The new combined architecture has provided us with a stable, faster operation of the systems and the almost complete absence of errors on some modules. It also served as the basis for the effective work of marketers. Thanks to the improved architecture, promotion specialists managed to increase the resource traffic by 20 times.
Microservices
In standalone microservices, we have allocated such modules as the file downloader, the search and the news feed. In addition to these understandable to the user and the obvious things, in the form of microservices many other functional elements were implemented:
In addition to all of the above, we have created many microservices for convenient work with the admin panel.
Each functional block of the site has become a fairly autonomous unit. A dedicated process is used for loading and rendering, minimizing intersection with other services.
Agree that when a user makes a payment, he will not want to wait long for the script to download, which is responsible for playing the video it does not need in another part of the page. Indeed, in monolithic systems this is the case — side functionalities sometimes impede the work of truly important components.
We took the first step and made life easier for both users and developers, releasing more than 20 new services in a year. As a result:
Technology stack upgrade
In parallel, we have updated the technology stack. If the portal was originally based on php 5.6 and MySQL 5.6, then in the course of improving the engine we translated it to php 7.0 and MySQL 5.7. Implemented framework Yii2, provided caching memcache.
Task queue
In order to better parallelize the tasks, we transferred the web workers - internal processors - to the RabbitMQ queue. It optimizes the sequence of launching events, which reduces the load on the system. Work with the site has become much faster and more comfortable.
Continuous integration
Transparent and manageable development process supported by GitLab CI. This is a really continuous seamless development, which leaves programmers and testers more time for important questions instead of patching holes.
In order to minimize errors, we introduced additional development contours. The chain consists of several contours for testing the application and the infrastructure: pre-release, where the latest build testing (pre-prod) takes place, combat (prod), several test contours for testing several builds at once, as well as a circuit for developers.
Build system
At the initial stage, builds are created from project sources. Instead of a separate set of classes, styles, and handlers, merged entities are used — each wrapped in a single executable file. The assembly contains only the necessary constructions for the execution of events on the page. This additionally provides the flexibility and speed of the site.
Error monitoring
GitLab CI gives you advanced code validation and automated testing. At the slightest non-compliance of assemblies with the specified parameters, the system notifies developers and does not allow starting the further process of component implementation before correcting errors. Additionally, we installed Sentry - a complete error monitoring tool.
Process optimization
To maintain the system in a state of high performance, we have consistently optimized business processes and infrastructure. Shared libraries and prototypes of services became the basis for the rapid introduction of new functions - we take ready-made templates and after a short time we get a working product.
Monitoring and operation
All processes - in full view. The monitoring of the status of each node is provided by Zabbix - a comprehensive tracking system. We got the opportunity to look into the very core of the project structure and see it compactly.
We also connected the Grafana platform, which turns dry data into a visual dashboard for the convenience of the customer’s employees.
Stabilization and refactoring
As required, we refactored the system — the code is simplified and concise. At the same time, there is always room for growth, since regular finishing of systems speeds up the site. As a result, the resource has become more search engine and user friendly.
New architecture - the basis for growth
At the start of the work, the resource had a modest potential for promotion and was not convenient enough for customers, which slowed down the company's business processes. To improve the process of user interaction with the site, we almost completely updated the portal architecture and its structure, created a flexible mechanism that integrates with a large number of services and applications, thanks to our own API. Many services have been working optimally for more than a year, without requiring intervention - the number of errors in their work has dropped to almost zero.
The new site is a powerful business development tool with potential for further scaling. Using the updated architecture of the web application, the project marketers were able to implement their range of promotional activities and increase traffic from 350,000 unique visitors per month to more than 7 million. At the same time, after a multiple increase in the audience, the portal continues to function stably.
- Moral obsolescence. The approach to usability, adaptation to different devices, elementary appearance and structure with time cease to respond to the needs of the audience.
- Technological obsolescence. Web standards and technologies are constantly being improved. Changes affect both developer tools (for example, frameworks, databases, other software), and the fundamentals of the web. From the relatively recent - a massive transition from http to https and the rejection of support for outdated DNS.
- Errors and failures. Developers may be imperfect to cope with their task and to allow flaws. As a result, violations of the site can occur - starting with the fact that the buttons do not work the first time, until complete failure of the systems.
Case
We were approached by a client whose web portal needed further development. At a certain point, a ceiling of 350,000 unique visitors was reached monthly. This seemed to be insufficient - the project clearly had prospects, and we had to figure out exactly what was in the way of further scaling the resource.
')
Audit
It turned out that the old Bitrix engine was not adapted to the loads and scaling. Every one of the components of the system closely intersected and depended on each other. Constantly eliminating errors was such a painful exercise that we proposed a flexible and understandable microservice architecture. In them, each component is sufficiently independent that the errors inside it do not “touch” the other elements of the system.
The basis of the architecture remained Bitrix - at least, authorization is tied to it, but where it was possible and promised greater efficiency, we transferred the functionality to microservices. They became the cornerstone of the project.
The new combined architecture has provided us with a stable, faster operation of the systems and the almost complete absence of errors on some modules. It also served as the basis for the effective work of marketers. Thanks to the improved architecture, promotion specialists managed to increase the resource traffic by 20 times.
Microservices
In standalone microservices, we have allocated such modules as the file downloader, the search and the news feed. In addition to these understandable to the user and the obvious things, in the form of microservices many other functional elements were implemented:
- Uploading, storing and managing data from open source projects;
- Dynamic creation of RSS feeds;
- Dynamic creation of a system of blocks, the layout of materials of any services into structural blocks;
- Adding additional data to service materials;
- Technical support service;
- Single catalog of various data.
In addition to all of the above, we have created many microservices for convenient work with the admin panel.
Each functional block of the site has become a fairly autonomous unit. A dedicated process is used for loading and rendering, minimizing intersection with other services.
Agree that when a user makes a payment, he will not want to wait long for the script to download, which is responsible for playing the video it does not need in another part of the page. Indeed, in monolithic systems this is the case — side functionalities sometimes impede the work of truly important components.
We took the first step and made life easier for both users and developers, releasing more than 20 new services in a year. As a result:
- Solved the problem of scaling.
- Implemented a more efficient development process by distributed teams.
- Set up dynamic resource management for smooth and fast system operation.
Technology stack upgrade
In parallel, we have updated the technology stack. If the portal was originally based on php 5.6 and MySQL 5.6, then in the course of improving the engine we translated it to php 7.0 and MySQL 5.7. Implemented framework Yii2, provided caching memcache.
Task queue
In order to better parallelize the tasks, we transferred the web workers - internal processors - to the RabbitMQ queue. It optimizes the sequence of launching events, which reduces the load on the system. Work with the site has become much faster and more comfortable.
Continuous integration
Transparent and manageable development process supported by GitLab CI. This is a really continuous seamless development, which leaves programmers and testers more time for important questions instead of patching holes.
In order to minimize errors, we introduced additional development contours. The chain consists of several contours for testing the application and the infrastructure: pre-release, where the latest build testing (pre-prod) takes place, combat (prod), several test contours for testing several builds at once, as well as a circuit for developers.
Build system
At the initial stage, builds are created from project sources. Instead of a separate set of classes, styles, and handlers, merged entities are used — each wrapped in a single executable file. The assembly contains only the necessary constructions for the execution of events on the page. This additionally provides the flexibility and speed of the site.
Error monitoring
GitLab CI gives you advanced code validation and automated testing. At the slightest non-compliance of assemblies with the specified parameters, the system notifies developers and does not allow starting the further process of component implementation before correcting errors. Additionally, we installed Sentry - a complete error monitoring tool.
Process optimization
To maintain the system in a state of high performance, we have consistently optimized business processes and infrastructure. Shared libraries and prototypes of services became the basis for the rapid introduction of new functions - we take ready-made templates and after a short time we get a working product.
Monitoring and operation
All processes - in full view. The monitoring of the status of each node is provided by Zabbix - a comprehensive tracking system. We got the opportunity to look into the very core of the project structure and see it compactly.
We also connected the Grafana platform, which turns dry data into a visual dashboard for the convenience of the customer’s employees.
Stabilization and refactoring
As required, we refactored the system — the code is simplified and concise. At the same time, there is always room for growth, since regular finishing of systems speeds up the site. As a result, the resource has become more search engine and user friendly.
New architecture - the basis for growth
At the start of the work, the resource had a modest potential for promotion and was not convenient enough for customers, which slowed down the company's business processes. To improve the process of user interaction with the site, we almost completely updated the portal architecture and its structure, created a flexible mechanism that integrates with a large number of services and applications, thanks to our own API. Many services have been working optimally for more than a year, without requiring intervention - the number of errors in their work has dropped to almost zero.
The new site is a powerful business development tool with potential for further scaling. Using the updated architecture of the web application, the project marketers were able to implement their range of promotional activities and increase traffic from 350,000 unique visitors per month to more than 7 million. At the same time, after a multiple increase in the audience, the portal continues to function stably.
Source: https://habr.com/ru/post/449538/
All Articles