Do not be afraid of microservice: Alexey Baitov on the use of microservice architecture in practice

For some, microservices are an opportunity to remake and refactor an application into a conditionally modern style. For others, this architectural solution is not suitable because of the peculiarities of interaction between different parts of the application. In any case, choosing the architecture, it is useful to study someone else's experience of moving from a monolith to a set of services.

We asked to share our case study of the development and delivery of microservices Alexey Baitov, a leading engineer of 2GIS. Let's talk about architectural solutions, deployment and scalability. Let's ask about trend and just convenient tools for work.



- Alexey, please tell us a little about yourself and about your team at 2GIS . What are you working on now?
')
I came to IT back in 2003 as a system administrator, plunged into development in 2011. During this time, I worked on PHP, JavaScript, implemented a series of RESTful services and a Python driver for Git. I have been working in 2GIS since 2015.

Participated in the development of two microservice architectures. The first consisted of one service. It was an asynchronous reverse proxy with a cache. In fact, he was engaged in sending messages. I worked on the requirements, the development and construction of DevOps alone, but I was helped by experts from our company 2GIS.

The service was written on Go. The quick compilation made it possible not to wait, and I was able to concentrate on Continuous Deployment. Then we just started to use GitLab CI , Prometheus , Grafana and Deis (open source analogue of Heroku ). We have the Infrastructure and Operations team, which just at the time of my development brought all these infrastructure solutions to production ready level. I decided to try all this and successfully implemented an independent microservice.

Two years ago, I moved to another team on a new project, where I became involved in functional programming on Scala. Our team from scratch has developed a microservice architecture for the storage of promotional materials 2GIS on Scala, C # and JavaScript. I laid the foundation of all the tools and experience for building DevOps (Continuous Deployment and Maintenance) practices. The architecture has gone from prototype to industrial exploitation. She swallowed two monoliths, now consists of 15 services and is constantly expanding.

- Do you agree that microservices are essentially a set of independently deployed services that have common characteristics, that is, it is a set of certain features that gives them the appearance of microservice? This definition needs to be expanded? Or, in fact, companies understand the microservice architecture differently?

I like the following definition . Microservice architecture is an architectural style that structures an application as a collection of loosely coupled services that implement a certain business logic. Services in the microservice architecture may not have common characteristics, but are combined within the framework of a common business logic.

Of course, each company will have its “own” microservice. This is a set of practices: distributed architecture, continuous integration and delivery, and so on. If you expand the concept of practice to the tools used, the implementation of microservices will be very diverse.

- There are different opinions about the composition of the team, which should be involved in the writing and support of microservices. What do you think about it? What is the optimal team size and how should the interaction be built inside it when developing a microservice architecture? Is there a good example of teamwork from your practice?

It is considered correct to develop a service so that its entire subject area can fit in the head of one developer. At the same time several people can participate in the development of this service. This will help avoid the bus factor when the developer went on vacation or fell ill. Correct splitting into services allows a new person to quickly enter the context.

"Microservice architecture" tells us that it often includes several services. Thus, one developer can not do. Microservice architecture is based on the product model (or general business logic). Developers are selected to implement this model and at the same time focus on the client.

Focusing on the client is organized by direct contact of the developer with the client. Developers need to see how their product is used. From this already follow the wishes for knowledge in the technological field, the ability to deliver the product to the client as soon as possible and to accompany the product.

It’s hard to say for sure about the number of teams. Everyone probably already knows the statement of Jeff Bezos, the founder of Amazon, that the size of the team for service-oriented development should be small enough so that everyone can be fed with two pizzas. In the comments on Habré there was a discussion on this topic, and there they wrote that one person may not have enough of one pizza and therefore the team should consist of one or two people. Martin Fowler, quoting a statement about two pizzas, said that we are talking about big American pizzas, after which he clarified that the team should not be 50, but around 12 people. I believe that everything depends on the product model. But the clarification of Fowler about "no more than 12 people" in my practice so far has won. I noticed that within the team it is desirable to divide by technological interests, find like-minded people.

It is not necessary that everyone in the team be well aware of all the technological areas used in the work, but the overall knowledge of the team must be uniformly deep. For example, two people are engaged in the initial building of deployment, and in the future, most likely, they will also significantly improve it. But at the same time, the whole team must have a good understanding of the deployment process. This will allow her to express wishes and make changes. Why two people? Because sometimes one person may fall into a creative stupor. And in the discussion truth is born.

We naturally built on this principle, united in technological interests. In this case, the developer can also engage in setting up a DevOps practice, and the QA engineer can develop an auxiliary, non-production service (for example, a cache warmer or anomalies search service in data in different environments).



- Almost every report on microservices begins with the story that “we had an iceberg and we sawed, sawed, sawed ... New parts of the application were made on the basis of microservices, and then they began to separate the“ pieces ”from the main engine ... "

Tell me, are you a supporter of development from scratch, or there may be situations when it is worth making a gradual withdrawal from a monolith? How to correctly determine the strategy of "exit"?

I am a supporter of development from scratch. But it works only in the case of a not too complicated set of functions. Usually a small MVP monolith is made. And sometimes you have to change its internal implementation several times. This can be caused both by a change in the technical task, and by the fact that an understanding of the implementation comes - high-level abstractions appear at the level of the business model. After that, you can go to the microservice architecture.

But if you work out these abstractions well at the very beginning and draw them in different notations (UML, BPMN, IDEF), so that all participants in the process understand what they are working with, then it is quite possible to implement the microservice architecture at once.

Our path to microservice architecture was not direct. At first there was a monolith. He processed text advertising materials. Three and a half years ago we needed to work with graphic advertising materials (images, logos). There was a desire to bring into business logic what was lacking when working with text advertising materials.

In order to try out a new business model, we implemented work with graphic advertising materials as a second monolith on another technology stack. After one and a half years of operation, we realized that this approach was correct.

During this time, we have a lot of hotelok, revealed the roughness of business logic.

The implementation of the second monolith was difficult to expand at first. Therefore, we decided not to conduct development in two monoliths at once, but to unite them within the framework of the third architecture for the very new business model. A team of seven developers, one QA engineer and two analysts was created. Two developers from this team previously created and maintained the first monolith, and another one the second monolith. That is, our team already at the entrance knew well the pitfalls of the previous monoliths.

The first monolith was written in C #. The second is in PHP. We did not want to lose the debugged large pieces of code from the first monolith and at the same time required multithreading, safe code and strong typing. PHP-code under these requirements partially did not fall. Therefore, C # remained as the basis and implemented what it did well in the first monolith - working with the content of promotional materials - but on the basis of another repository: S3-compatible storage and Kafka.

This time, Scala and the PostgreSQL database were chosen to work with the very new business model. Scala met our specifications. In addition, Scala developers were located on the same floor as C # developers. This reduced the time for inter-team communications. Conway's law worked - the structure of the company dictated the structure of the application. The PHP developer has reassigned to a Scala developer. I just finished working on an independent microservice on Golang with a full CI / CD cycle, after which I joined the team and also became a Scala developer.

It is interesting that I suggested using Scala instead of C # to work with business logic. The fact is that we didn’t have enough C # developers. PHP-person and I wanted to retrain on Scala. Plus, we had the opportunity to attract an experienced Scala developer. Another point: if we implemented everything in C #, then we could get either a microservice architecture or another monolith at the output. The division into Scala and C #, different storage needs and the presence of experienced developers in each of the required areas - all this directly pointed us to the microservice architecture. And we only benefited from it. A year ago, the micro-service architecture for working with graphic and textual materials went into commercial operation and is still working successfully.

On the question of whether it is possible to create a microservice architecture from scratch. A year and a half ago, in the process of working on microservice architecture, we were asked to support a new direction in our products - video advertising materials. It was necessary to test a new sales model in a short time. Our architecture was in its infancy. Work with video materials covered a new area of ​​technology. We decided not to change the development vector and implemented the MVP on video as a stand-alone microservice architecture in C # and trusted video hosting. This is a successful experience, and we have a report on this topic. Thus, we have two parallel microservice architectures. MVP didn’t develop much, Wishlists also accumulated on it and soon we will unite everything within the framework of a single microservice architecture - we’ll have a single repository of advertising text, graphics and video.

- Immediately, there are two important factors that speak in favor of microservices. The first is the ability to output individual parts to the cloud and, as a result, colossal scalability. The second is the ability to create a separate service in another language. What else do you see the advantages of switching to microservices? Well, about the minuses, of course, I also want to hear.

If we talk about the technological component, then the advantages, in addition to the above, include the possibility of using another stack of technologies. And if he does not suit us, choose another one. New technologies bypass the problems of old solutions. Microservice architecture also gives stability and independence: degradation of one service should not lead to complete degradation of the entire system. The composability of services allows re-use of service functionality in other microservice architectures. From the point of view of the organizational component, the division into services allows to divide the development within distributed teams or one large team.

The main disadvantages of microservice architecture: it is much more complicated, and its implementation is more expensive.

You also need to be prepared to support service contracts, choose the right remote access protocol, resolve issues of secure interservice communication, possible failures, as well as deduplication and management of distributed transactions.

In general, in the public domain you can find a lot of information and materials on how to work with it. But in fact it all depends on the task. In my practice, the pros have always been more significant than the minuses.

Another thing to remember is Sam Newman's words: the smaller the service, the more pronounced all the advantages and disadvantages of microservice architecture.

- You have some interesting reports about microservices. Deploying microservices and the Approach to Continuous Deployment in microservice architecture . In one of the slides of the first one, there are “templates” of deployment, and in the report you say that the trend approach for you is distribution through containers. During this time, nothing has changed? Bundle Docker + Kubernetes has not lost its relevance?

We are transferring more and more services to this bundle. I think that we have chosen the right direction and while we are planning to stick to it. If something changes, I will tell you about it in a new report or interview.

- How trouble free is the continuous deployment and transfer of microservices to the prod?

It all depends on how you build the process. At first it seems that everything is simple. Services are independent, deployed separately. Weak connectivity is provided by different approaches to work with the evolution of the contract. And here you have to choose. For example, you can implement contract versioning or add a service to terminate a contract (contract decoupling).

If 10 or more services are in active development in the microservice architecture and everyone has their own versioning, then the problem of version consistency arises. We must try not to get confused in the compatibility of services of different versions.

Continuous deployment means that we can deliver the application at any time to any environment.

Application in microservice architecture is a collection of services. So, at any time we need to know a stable combination of versions of services. And we must somewhere have a set of domain addresses and other parameters specific to different environments for configuring services and associating them with each other.

All this needs to be stored somewhere, edit changes in several places (microservices are independent) and not be confused.

Continuous deployment implies the ability to roll back to any version at any time. Accordingly, it may be that you need to roll back several services at once and you will need to observe the correct reverse order of the service deployment.

In one of my reports, I talk about our approach to the evolution of contracts, about how they solved the problem of consistency of versions and how they built up the deployment process in the micro-service architecture from more than ten services. Continuous deployment can not be trouble-free, but everything can be solved.

- What could be the initial set of tools for continuous deployment of microservices? What options would you recommend to use for working with microservices?

Continuous deploy is a sequence of stages (pipelines) that include continuous integration, integration tests, and service delivery to the orchestration environment. The most popular tools for successive stages are Jenkins 2 Pipelines , TeamCity Build Chains , Bitbucket Pipelines and GitLab CI Pipelines . First you need to automate continuous integration (Continuous Integration, CI). We need a remote CI server that would build and run tests on this build.

The listed tools offer their solutions. We use GitLab CI, and GitLab Runners act as such servers. The build artifact is a Docker image. As part of the integration tests, it is possible to carry out load and capacitive tests using Gatling , in particular, in order to determine the resources it needs (processor and memory) for operating on the orchestration environment (for example, Kubernetes ). Helm is widely used for deployment, it allows you to describe microservice architecture for different environments. In our company we do not use Helm. We have our own deployment tool, which was created when Helm was in the Classic version and did not support different environments. Both of these tools have similar useful qualities, but the implementation and distribution are different. And our own tool allows you to make improvements to the deploy and adapt everything to our infrastructure.

- What technologies are optimal for small and medium-sized companies that want to implement microservices? Is it too expensive for them?

I increasingly find confirmations that it is inappropriate for small and medium-sized companies to raise their own orchestration environment (for example, Kubernetes , Docker Swarm or Apache Mesos ). After all, for this they need to contain their hardware and one or more commands to configure and support it. It is easier to use popular third-party cloud services (based on the Google Cloud Platform , Amazon Web Services , Microsoft Azure or Oracle Cloud ) and choose the right tariff.

We use the free version of GitLab and GitLab CI. This tool allows you to store the code and see all the information about its deployment in one place. GitLab is actively integrating with Helm and various cloud services. I often say Helm and do not bring analogues. Helm has its drawbacks, but these drawbacks are different for everyone and depend on the scale of development, so you can start with it, and then you will definitely find the necessary tools in addition.

I would also recommend our configuration and deployment tool, which I mentioned earlier, but it is not yet available as open source, although work is actively being done in this direction.

It is also interesting to look at Spinnaker and Heroku - these are different, but proven solutions that allow you to quickly get stuck on cloud services.

- On the one hand, version control is needed to ensure communications control and reliability at a level sufficient for industrial operation. Moreover, with a large number of services, versioning can seriously slow down the development of infrastructure. How did you solve this problem?

Developing microservice architecture from several services, we have gone from a prototype to industrial operation. We first encountered versioning when we needed to close a unified test environment from all services, and then, in the course of changing each service, maintain the stability of the existing functionality.

We didn’t want to produce versions (and there are a lot of them at the prototype stage) or introduce different services for contract separation. Microservice architecture allowed replacing one service with another with another implementation or on a different technology stack.

When adding or removing services, versioning does not help. We have proposed our approach to the evolution of the contract. We combined the most frequently changing services into a single cast of versions, also known as a package. Then a simple package manager was implemented and they began to use this cast for deploying a collection of frequently changing services.

The versions of services (not contracts) are Docker image tags, which in turn are equal to hash commits or tags in Git. We had to rebuild the continuous deployment, but this only made it easier. I talked about this in detail in the report, and the dynamics of deployment can be seen on the slides . In general, we were able to abandon the usual versioning of the contract at the prototype stage and significantly accelerated the development.

We added the versioning of the contract to services one month before it went into commercial operation, when we needed to integrate with external customers. And then: we added it only to the public API. And now we have been in industrial operation for a year and during this time we have raised only three minor versions. At the same time, we continue to engage in active development: we add and remove functions. We have client services that use our public API. We know which API methods external clients use and which ones we use. If the changes concern only our methods, then in most cases we do not raise the API version, change only the version cast and do not make irreversible changes over short periods of time.

Thus, we were able to abandon the many versions of the contract. If we left them, we would have to test each version; ask external customers to upgrade to a new version of the public API, although most of the changes do not concern them; track whether the old version is being used and remove it from the code.

I know that you are using an approach in which part of microservices is divided into separate groups. Can we say that they become an analogue of the application kernel?

The core is something internal. If you look at it as a multi-layer microservice architecture, where there are layers with services for a high-level API, integration services and small atomic services, then the core in this case are just these low-level atomic services. We have another division. In a separate group, we allocate frequently changing services on different layers of the microservice architecture.

Services that change rarely have their own independent process of continuous deployment. For the deployment of a separate group, a separate repository was created, where deployment configurations are stored, there is a package manager for the version cast and a continuous deployment for different environments. At the same time, continuous integration, which is the basic part of continuous deployment, is distributed over repositories of frequently changing services.

- It is always interesting to learn not only about the approaches and technologies, but also about the personal experience of companies. Let's take 2GIS. Say, the process of transition to microservices took a lot of time? What is finished now?

The transition began more than three years ago. At first, we conducted a study. We tried Mesosphere , OpenShift and other PaaS and IaaS solutions available at that time. The choice fell on Deis and we have a report on this topic, as well as a report on the support of Deis for one and a half years. This is an open source analogue of Heroku , it allowed to deliver its service as Heroku Buildpack, Dockerfile or prepared Docker-image. He had good documentation and competent architecture. We wrote a family of make files as a universal set of commands on top of the command console for deploying to Deis. This was the first implementation for continuous deployment.

Next, the company moved to Deis2 . He was backward compatible with Deis, but Kubernetes was already used as an orchestration medium. The Infrastructure & Operations team , which was engaged in raising these platforms and their further maintenance, developed its tool for deployment to Kubernetes and Deis2. Those who switched to microservice architectures, in most cases, chose Deis2, but there were also those who chose the low-level path - direct work with Kubernetes. The latter were right. Deis2 had a number of flaws: it was not possible to go into a running container, the pod could consist of only one container, and for each pod, its own namespace was created. This did not give flexibility and complicated the work of the Infrastructure & Operations team. Kubernetes was devoid of these shortcomings. As a result, they decided to transfer all services from Deis2 to Kubernetes. A full rejection of Deis2 in favor of Kubernetes is scheduled for this summer.

- Tell me, if you were now to rewrite everything under the microservice architecture, having already had this experience behind you, what would you change in the implementation? What interesting ideas are already impossible to apply?

I would immediately read about Helm. It has very large documentation and is hard to read in a short period of time. But Helm builds a good understanding of the practice of deployment of microservice architecture.

Helm immediately suggests that the microservice architecture is a collection of services: although they are weakly connected with each other, they are deployed together.

In Helm there is a dependency of charts and you can describe a root chart that would describe the services and link them according to the parameters inherent in a certain environment.

The deployment tool developed in 2GIS allows you to combine multiple services at once, but it is great for a collection of no more than 10 services. We had more than ten services (on test environments for each task we raised additionally our backing services: Postgres, Kafka, Zookeeper, Ceph). At some point, I realized that yaml programming does not reveal the full potential, it is inconvenient to work with it in the IDE, aliases and anchors do not cover the entire nesting. I realized that I need a programming language. I chose Python because a deployment tool was written on it and Python is definitely in the container with this tool. Described an abstract factory that would generate a family of service descriptions. This allowed us to add descriptions for abstractions that we do not deploy. For example, the parameters of access to third services. They were still represented by classes, like the descriptions of our services, and were still generated by an abstract factory. Now you can put some descriptions in others and reuse descriptions. You can inherit environments, for example, load from steding, because you would like to have the same actual description of resources (processor and memory) for both environments. I realized that I had gone farther than Helm, and I am interested in solutions for describing Kubernetes resources in some programming language, and not on yaml.

I would also make the public API split into several services and implement the API Gateway . But the current implementation copes with the task, and the best is the enemy of the good.

But whatever I definitely did not change, it is the result of a choice: a monolith or microservice architecture. Microservice architecture brought us much more advantages than the difficulties that we were able to successfully overcome.

The principles of DevOps penetrated tightly into the lives of developers, it’s time to get together at a large professional conference to share experiences. Join DevOpsConf Russia .

DevOpsConf Russia will take place on October 1 and 2 in Moscow and will logically continue the RootConf series, the name of which, however, has ceased to reflect what we are doing. We are going to discuss DevOps as a unit from all development, testing and operation processes.

If your company DevOps is gaining momentum, or the transformation has just begun, but there is already some success - come share them with the community. The program committee is waiting for your applications until August 15.