JARVIS - the invisible assistant to Leo
Sooner or later, IT projects face difficulties in maintaining high code quality and / or increasing the delivery time for changes in production. Lingualeo has experienced all the problems of growth and is ready to share its history of improving the efficiency of development. Mikhail Kabishchev told teamlead about how it happened .
Like any other technology company, Lingualeo went through several stages:
As a result, routine operations begin to take a lot of time, and the company is thinking about how to automate these processes.
Intuitively, the Lingualeo team quickly understood that it was time to move on to the next stage. Sometimes the transition took place in the right direction, and sometimes not. At some point, several development teams formed in the company that worked in one common system. Each team had its own project in jira, its own CI system (Continuous Integration system), which was able to run unit tests, deploy platforms for testing, build and deploy releases.
')
Sounds good, right?
But still, there were moments in this approach that we did not really like:
A large number of projects in jira. Initially, all used one workflow. But then one team added an additional field, another - an intermediate status, and so on. Because of this, constantly had to make changes to the CI system, and sometimes these changes conflict with each other.
Samopisnaya CI-system. She was cool, completely solved the tasks assigned to her until there were a lot of developers, they felt free and began to actively create new repositories. The system was not ready for this. The assembly had to wait a very long time, the viewing of logs was not implemented in the most successful way, but also the tasks for assembling releases hung in the common queue.
Depla. Deployment mechanism was sharpened for the release of only one application. At the same time we had a need for several.
At the next retro, we realized that it was impossible to continue living like this, and put forward several theses:
Considering our previous experience in designing a new workflow, we decided to divide all possible types of tasks into two groups:
These are tasks that do not require writing code and have the simplest flow:
The yellow color on the diagram shows the steps that need to be performed manually, and the blue ones that need to be automated.
Such tasks already require writing code in one of the repositories. As a framework for working with code, we use gitflow , so the separation is as follows:
The purpose of the types can be found in any description about gitflow. We divided the feature and bug into separate types, so Lingualeo is more convenient to prioritize tasks and conduct analytics. Previously, each team had its own agile-board with its own tasks, it was very convenient, and it was necessary to think of a way to separate tasks into teams.
We also had some minor problems with the use of the assignee field: with some transitions it changed, with some it did not, and sometimes it changed for the wrong person. We decided to introduce a hard and fast rule: if your name is in the assignee field, then you are responsible for this task at this moment. Here is a list of interesting fields that we began to actively use:
In the same way as everyone, the Lingualeo team looked at the three main players in this market: Jenkins, Teamcity, Bamboo. We tested all three, but most of all I liked Teamcity: free, there is a convenient REST API and a nice interface.
So, we got the following configuration: jira — for running tasks, teamcity — for running tests, building releases, etc., and github — for storing code. We needed to make friends all these systems together.
For each pair there are several plug-ins, but they all seemed to us either not very convenient, or did not provide the necessary functionality. Therefore, we decided to write a small kernel that will manage all systems.
When designing, we took into account the requirement “to replace any component of the system painlessly and with minimal effort”. For example, to replace the build-server, we only have to write a new adapter, and not to look for a bunch of plug-ins.
We wanted JARVIS to be able to do the following operations for us:
In addition, we wanted as little as possible to make changes to the source code of the system, it should provide us with blocks (actions) from which we will assemble our processes, and we decided to describe the sequence of actions and the conditions under which to perform them, as yaml- files. Here is an example configuration that describes the rules for running code verification:
The first rule finds all tickets with the Ready for Inspection status and runs the appropriate configuration in the teamcity, which checks the code for compliance with the standards, and also runs all the unit tests. If the launch of the configuration is successful, then the start inspection transition is applied to the ticket, and it switches to the On Inspection status. The second rule checks all running configurations. If it is successfully completed, then the ticket goes on. If errors occurred during the build, then the ticket through the Fail Inspection transition goes back to the developer.
Lingualeo uses this scheme not only for the main repository, but also for all internal libraries and for mobile applications. In this case, the creation of test benches and deployment is replaced by the assembly of test and production versions of applications. Adding a new repository takes only a few minutes, and for it we get automatic code quality check, review creation, build builds and release.
JARVIS itself is developed and built with JARVIS `a :)
Due to the unification of the development process in the company and the creation of JARVIS, we were able to improve the quality of the code we produce and reduce the delivery time for changes in production, reduced the time of developers, which was spent on routine operations.
Like any other technology company, Lingualeo went through several stages:
- Start of product development. Development and debugging occurs on a single server, where everything that the project needs is running. Errors are often, but it's not scary, because it's all just a prototype, and there are no live users there yet.
- The appearance of the first users. The company is starting to feel the price of mistakes and problems in production. Already it is impossible to rule everything on production, there comes an understanding of what needs to be thought of releases. Developers implement workflow to work with the code base, something like stage-server appears on which releases are tested.
- Growth of the project and team. A large number of tasks are under development at the same time. Requirements for the process and quality of the code are greatly increased. It’s very hard to follow everything: someone forgets to run unit tests, someone doesn’t know where and how to close another task for testing.
As a result, routine operations begin to take a lot of time, and the company is thinking about how to automate these processes.
Intuitively, the Lingualeo team quickly understood that it was time to move on to the next stage. Sometimes the transition took place in the right direction, and sometimes not. At some point, several development teams formed in the company that worked in one common system. Each team had its own project in jira, its own CI system (Continuous Integration system), which was able to run unit tests, deploy platforms for testing, build and deploy releases.
')
Sounds good, right?
But still, there were moments in this approach that we did not really like:
A large number of projects in jira. Initially, all used one workflow. But then one team added an additional field, another - an intermediate status, and so on. Because of this, constantly had to make changes to the CI system, and sometimes these changes conflict with each other.
Samopisnaya CI-system. She was cool, completely solved the tasks assigned to her until there were a lot of developers, they felt free and began to actively create new repositories. The system was not ready for this. The assembly had to wait a very long time, the viewing of logs was not implemented in the most successful way, but also the tasks for assembling releases hung in the common queue.
Depla. Deployment mechanism was sharpened for the release of only one application. At the same time we had a need for several.
At the next retro, we realized that it was impossible to continue living like this, and put forward several theses:
- All development should “move” to a single project in jira with one for all workflow.
- You can use a ready build server instead of a self-made one.
- Deploy system should be universal to deploy different applications.
Single project in Jira
Considering our previous experience in designing a new workflow, we decided to divide all possible types of tasks into two groups:
Task
These are tasks that do not require writing code and have the simplest flow:
- Task. Just a challenge. This could be research, adding a new repository, writing documentation, etc.
- Migration. We perform all migrations for the database manually. Manually does not mean that we do it directly in the sql console, there are a number of tools for this, but they are started manually.
CI unit
The yellow color on the diagram shows the steps that need to be performed manually, and the blue ones that need to be automated.
Such tasks already require writing code in one of the repositories. As a framework for working with code, we use gitflow , so the separation is as follows:
- feature
- bug
- hotfix
The purpose of the types can be found in any description about gitflow. We divided the feature and bug into separate types, so Lingualeo is more convenient to prioritize tasks and conduct analytics. Previously, each team had its own agile-board with its own tasks, it was very convenient, and it was necessary to think of a way to separate tasks into teams.
We also had some minor problems with the use of the assignee field: with some transitions it changed, with some it did not, and sometimes it changed for the wrong person. We decided to introduce a hard and fast rule: if your name is in the assignee field, then you are responsible for this task at this moment. Here is a list of interesting fields that we began to actively use:
- Team. The name of the team that owns the task.
- Component. In our case, the component uniquely points to the repository. Although jira allows you to specify several values ​​in the component field, we have no problems with this.
- FixVersion. The release number to which the task falls.
- Reporter. The person who created the task.
- Customer. This is a customer task. By default, it is equal to reporter, but can be changed. It is he who performs the task acceptance.
- Developer. The developer responsible for the task. A task can have subtasks that will be done by different people, but it is developer who is responsible for the task entirely.
- QA Engineer. By analogy with developerʻom, this person is responsible for testing the task entirely.
- Code reviewer. All tasks (well, almost all) we have a code-review, and this is the person who does it.
Build server
In the same way as everyone, the Lingualeo team looked at the three main players in this market: Jenkins, Teamcity, Bamboo. We tested all three, but most of all I liked Teamcity: free, there is a convenient REST API and a nice interface.
So who is JARVIS?
So, we got the following configuration: jira — for running tasks, teamcity — for running tests, building releases, etc., and github — for storing code. We needed to make friends all these systems together.
For each pair there are several plug-ins, but they all seemed to us either not very convenient, or did not provide the necessary functionality. Therefore, we decided to write a small kernel that will manage all systems.
When designing, we took into account the requirement “to replace any component of the system painlessly and with minimal effort”. For example, to replace the build-server, we only have to write a new adapter, and not to look for a bunch of plug-ins.
We wanted JARVIS to be able to do the following operations for us:
- check code for compliance with standards
- run unit tests
- create and pull pull request `s in github`e
- create test benches for testing tasks
- create release and hotfix branches
- produce warm
In addition, we wanted as little as possible to make changes to the source code of the system, it should provide us with blocks (actions) from which we will assemble our processes, and we decided to describe the sequence of actions and the conditions under which to perform them, as yaml- files. Here is an example configuration that describes the rules for running code verification:
start inspection: search: jql: 'project="DEV" AND status="Ready for Inspection"' action: type: 'run-build' params: buildTypeId: 'CodeInspection_%component%' success: transition: 'start inspection' fail: transition: 'fail inspection' complete inspection: search: jql: 'project="DEV" and status="On inspection"' action: type: 'check-build' params: buildTypeId: 'CodeInspection_%component%' success: transition: 'complete inspection' fail: transition: 'fail inspection' The first rule finds all tickets with the Ready for Inspection status and runs the appropriate configuration in the teamcity, which checks the code for compliance with the standards, and also runs all the unit tests. If the launch of the configuration is successful, then the start inspection transition is applied to the ticket, and it switches to the On Inspection status. The second rule checks all running configurations. If it is successfully completed, then the ticket goes on. If errors occurred during the build, then the ticket through the Fail Inspection transition goes back to the developer.
Lingualeo uses this scheme not only for the main repository, but also for all internal libraries and for mobile applications. In this case, the creation of test benches and deployment is replaced by the assembly of test and production versions of applications. Adding a new repository takes only a few minutes, and for it we get automatic code quality check, review creation, build builds and release.
JARVIS itself is developed and built with JARVIS `a :)
Due to the unification of the development process in the company and the creation of JARVIS, we were able to improve the quality of the code we produce and reduce the delivery time for changes in production, reduced the time of developers, which was spent on routine operations.
Source: https://habr.com/ru/post/278293/
All Articles