Continuous Delivery in Yandex. How to overclock your development cycle using only Open Source solutions
There were always two tasks before testing - to help the team maintain a high level of development quality and do it without delaying the whole process. And this is true not only for our projects in Yandex, where we are working on a very large number of services. Often the main task is altogether formulated as an increase in the speed of testing while maintaining an adequate level of quality. The speed of the development process, commitment to the values ​​of frequent and quick releases - these are fundamental factors for the success of any product. The team has more opportunities for maneuver, the team quickly finds and corrects errors, gets feedback quickly. How to accelerate without losing quality, how to achieve a zen of continuous change delivery?
Today we show that Continuous Delivery is easy and fun! And the benefit from it can be obtained by incorporating it even partially. We have been using a similar approach for our open source libraries, Allure Framework or Yandex QATools, for several years in testing Yandex. The process is simple, significantly scalable and can be used for huge teams of one person, and for small teams of dozens of people. And most importantly - the entire toolkit is available in Open Source!
')
By the way, before September 30, you can apply and enroll in our School of automation of development processes in St. Petersburg. Training in it is free and will consist not only of a course of lectures - an obligatory stage will be team work on an educational project.
Now back to the topic. Imagine the picture: a cozy workplace, you write code, add unit tests and send changes to the version control system, and after a couple of hours they “go” to the combat servers. And everything works.
It seems that such a process is characteristic either for the development method of “chloby, khloby and in production” (by removing part of the unit tests), or for the commercial of a company specializing in innovation in software development. At the same time, a running line is required under it, similar to the signature under stunt tricks: “Danger, do not try to repeat at home!”.
The history of one developer . I heard a backup task somewhere: “How many trucks do you need for the administrator to start making backup copies?”. Humor is black, so the decision is waiting in the comments. Here we are talking about the development process, but the essence does not change. The developer sends any change to the main branch and, considering that it is time to assemble the package for testing and display, stops at a stable, in his opinion, state of the code, updating the branch from which the packages are collected. Now the question arises: "What is contained in the new package after the next assembly?". Any answer to this question may be wrong. After all, the developer is sure that only he changes the code, so it’s not necessary to look at the change history. It used to be, literally a miracle, found when testing anomalies that could seriously complicate the lives of the users of our services. Anomalies turned out to be inconsistent changes, which fell into a branch either from a developer from another project who decided to fix the bug, or collected “for the company” and simply forgotten. Difficulties expected and those who would decide to join the development process. Which branch is being developed? In which branch is the stable version? How to build a package?
And if there were errors in the package, after their revisions the tester could wait for a surprise - along with the corrections several more new features could “arrive”. With a lot of luck after a couple of cycles, the package could increase the initial number of tasks several times and “swell” to unimaginable sizes.
The story of several developers . This story differs only in that in the list of the daily routine, each developer in the team is added the merging of their changes with the changes of their colleagues. And after all, everyone trusts each other and no one looks at these very changes of colleagues - they know what they write! Everyone would be happy to see what is happening with the project, especially when a colleague goes on vacation, you still have to figure it out, but where to do it? There is nowhere to leave a comment in the development environment. Personally, it is not always convenient to say, especially if the development is carried out in different offices. And the changes are already in the project - it’s so easy not to redo it in case of errors.
What do these stories have in common? Excuses. Everyone believes that it is too difficult to simplify your life by entering an understandable process and automating the routine. In addition, everyone will have to break themselves on the knee to get used to the new realities. This is not true. We have already destroyed such a myth in our teams.
It is unproductive to rush to destroy harmful, but insignificant myths, about which no one has heard or from whom nobody suffered. Therefore, having experienced the first difficulties with delivering changes to the final consumers, the current development scheme had to be changed.
For total fun, you will need: GitHub Flow as an immediate process, CI server. A bit, right? Our libraries live on GitHub, so I will use its terminology, giving comments along the way. This whole process is quite possible on almost any combination of CI servers and popular platforms for IT projects, such as GitLab , Bitbucket . The key is only one thing: you need a glue called the Pull Request Builders extension family.
Fork, commit, push, request, build, review, merge, release.
I will not use all these funny words anymore. Though very pulling! The one who knows them and regularly uses, most likely, introduced a similar process long ago. These short incomprehensible words are as easy to learn as they are to implement the following basic steps as part of the task of easy process construction:
The entry point is the main repository, the master branch. One of the main benefits of the approach is always known where the stable working code is located, which was seen not only by the developer. Direct modification of this branch is prohibited. Even if you really want.
At this stage, only GitHub or equivalent is required. The first thing to do is to duplicate the current master branch of the main repository to your personal repository. Here the developer is free to build the encoding process as he likes. This is his personal repository, and here he is the absolute master. The ultimate goal is simple - a set of changes, based on the master branch of the main repository, in any branch of the personal one.
At this stage, the biggest challenge is getting used to writing unit-tests for your changes. The most interesting thing is that the difficulty is not to write these tests, but to train yourself to do them. But if you develop such a habit, it will pay for almost any effort on the writing of tests. Start small: cover positive scenarios. They are easy to think through, easy to implement in tests. Having got used not to be afraid for breakage of the declared functionality, it is easy to add negative scenarios. This will eliminate the fear of irreparably breaking in unexpected places. Are the changes ready for the demonstration?
On the set of possible cookies and implementation options, this is the most technologically advanced stage. The stage is not just called “watching others” - without the word “developers”. Automation!
To get started, the developer needs to start the process - to offer his changes to the merge into the main branch of the project. I think it is not necessary to say that telepaths are usually away, so the messages for each change should clearly reflect their essence. This is especially felt when you come back after a break to work on a project and discover other people's changes. Well, if it is clear what happened on the headings in the history of GIT. After all, after a vacation, I don’t really want, looking at the difference, to digest tons of changes to make a small correction. In open source projects, this is more than important - here hundreds of people are already watching your story!
This is where the CI server comes in. We use two: TeamCity & Jenkins . What exactly to use, we have not yet decided. It depends mainly on the preferences setting up a new build task. In order for the assembly to begin, as soon as the merge proposal arrives in the main repository, you will need the same glue — one of the extensions of the Pull Request Builders family.
For example, for Jenkins + GitHub, this is GitHub PRBP .
For Jenkins +, Bitbucket is a Bitbucket PRBP .
For Teamcity + Github, this is TeamCity.GitHub .
The CI server will carefully collect the proposed change, run the tests and be sure to report problems in the merge proposal itself:
Right here you need to configure and static code analysis. Now you can automatically find a huge number of errors in the code! For example, using the SonarQube . He will be able to measure the test coverage and warn you if it suddenly drops. Getting feedback takes minutes! You can correct errors in the same branch from which there was a sentence:
Do not be afraid to completely overwrite the story in this thread. GitHub, for example, will quickly update the sentence, and the CI server will re-compile it again. In many open source projects, such a rule is separately stated.
After successful assembly - time to inspect the code a living person. Of course, if you are the only developer, you may have no one to ask to see your code. In this case, it is an excellent chance to take a critical look at the changes made. After the assembly has been completed and the code is scanned, it remains only to press one button - upload the changes to the main branch. Responsible step. All systems reflect who did it.
With small changes, reading this section takes much more time than what it is written about.
As soon as the changes hit the main branch, the time comes for integration testing. Here it is most convenient to put the task of assembling test packets on guard. We at our moment are updating the SNAPSHOT version in the Maven repository in our libraries. There are already chasing heavy integration tests that test interaction with third-party components, browser tests are run, where it is required. At this stage, it is quite possible to give the packages in manual testing.
As soon as a sufficient number of changes have accumulated in the main branch, with which all participants are satisfied to the extent necessary, the time for release comes. Usually this is another task for the assembly with a manual call. In GitHub, for example, use the command for HuBot for this. It is very convenient to mark the head change with a tag at this moment. This will allow at any time if you want to be able to build the desired version without remembering the change number.
Choosing a release version is a separate process for which even specific standards exist. For small libraries, we use two-digit notation: the youngest grows each release, the older one, with the loss of backward compatibility, resets the younger one. For larger libraries - three-digit with the same logic.
In order to organize such a process, you do not even need to have your own CI server. For small projects it is very convenient to use www.cloudbees.com/jenkins , which will give a full-fledged Jenkins server with a limit on the number of assemblies. Starting to build merge offers on GitHub is quite realistic using buildhive.cloudbees.com - it has no limits. Or travis-ci.org , for which you need to put a small configuration file in the root of the project.
If you have your server at hand, set up Jenkins or TeamCity on it - it's half an hour. The default settings in these servers cover 80% of basic needs, and you just need to add the necessary extensions.
And having set up a complete chain on a small project once - it is possible to scale a solution to neighboring projects of even larger sizes in a couple of hours from absolute zero.
If you are interested in doing such things, it is interesting to build automatically scalable systems that facilitate development, assembly, and most importantly, testing, if it is interesting to improve the quality of products used from small to large, welcome to our School of Development Automation !
Today we show that Continuous Delivery is easy and fun! And the benefit from it can be obtained by incorporating it even partially. We have been using a similar approach for our open source libraries, Allure Framework or Yandex QATools, for several years in testing Yandex. The process is simple, significantly scalable and can be used for huge teams of one person, and for small teams of dozens of people. And most importantly - the entire toolkit is available in Open Source!
')
By the way, before September 30, you can apply and enroll in our School of automation of development processes in St. Petersburg. Training in it is free and will consist not only of a course of lectures - an obligatory stage will be team work on an educational project.
Now back to the topic. Imagine the picture: a cozy workplace, you write code, add unit tests and send changes to the version control system, and after a couple of hours they “go” to the combat servers. And everything works.
It seems that such a process is characteristic either for the development method of “chloby, khloby and in production” (by removing part of the unit tests), or for the commercial of a company specializing in innovation in software development. At the same time, a running line is required under it, similar to the signature under stunt tricks: “Danger, do not try to repeat at home!”.
As it was before
The history of one developer . I heard a backup task somewhere: “How many trucks do you need for the administrator to start making backup copies?”. Humor is black, so the decision is waiting in the comments. Here we are talking about the development process, but the essence does not change. The developer sends any change to the main branch and, considering that it is time to assemble the package for testing and display, stops at a stable, in his opinion, state of the code, updating the branch from which the packages are collected. Now the question arises: "What is contained in the new package after the next assembly?". Any answer to this question may be wrong. After all, the developer is sure that only he changes the code, so it’s not necessary to look at the change history. It used to be, literally a miracle, found when testing anomalies that could seriously complicate the lives of the users of our services. Anomalies turned out to be inconsistent changes, which fell into a branch either from a developer from another project who decided to fix the bug, or collected “for the company” and simply forgotten. Difficulties expected and those who would decide to join the development process. Which branch is being developed? In which branch is the stable version? How to build a package?
And if there were errors in the package, after their revisions the tester could wait for a surprise - along with the corrections several more new features could “arrive”. With a lot of luck after a couple of cycles, the package could increase the initial number of tasks several times and “swell” to unimaginable sizes.
The story of several developers . This story differs only in that in the list of the daily routine, each developer in the team is added the merging of their changes with the changes of their colleagues. And after all, everyone trusts each other and no one looks at these very changes of colleagues - they know what they write! Everyone would be happy to see what is happening with the project, especially when a colleague goes on vacation, you still have to figure it out, but where to do it? There is nowhere to leave a comment in the development environment. Personally, it is not always convenient to say, especially if the development is carried out in different offices. And the changes are already in the project - it’s so easy not to redo it in case of errors.
What do these stories have in common? Excuses. Everyone believes that it is too difficult to simplify your life by entering an understandable process and automating the routine. In addition, everyone will have to break themselves on the knee to get used to the new realities. This is not true. We have already destroyed such a myth in our teams.
It is unproductive to rush to destroy harmful, but insignificant myths, about which no one has heard or from whom nobody suffered. Therefore, having experienced the first difficulties with delivering changes to the final consumers, the current development scheme had to be changed.
Tools
For total fun, you will need: GitHub Flow as an immediate process, CI server. A bit, right? Our libraries live on GitHub, so I will use its terminology, giving comments along the way. This whole process is quite possible on almost any combination of CI servers and popular platforms for IT projects, such as GitLab , Bitbucket . The key is only one thing: you need a glue called the Pull Request Builders extension family.
Salt in 5 seconds
Fork, commit, push, request, build, review, merge, release.
Salt in three stages
I will not use all these funny words anymore. Though very pulling! The one who knows them and regularly uses, most likely, introduced a similar process long ago. These short incomprehensible words are as easy to learn as they are to implement the following basic steps as part of the task of easy process construction:
- developer makes change;
- others are watching the change;
- The change falls into the main branch of the version control system.
Developer makes change
The entry point is the main repository, the master branch. One of the main benefits of the approach is always known where the stable working code is located, which was seen not only by the developer. Direct modification of this branch is prohibited. Even if you really want.
At this stage, only GitHub or equivalent is required. The first thing to do is to duplicate the current master branch of the main repository to your personal repository. Here the developer is free to build the encoding process as he likes. This is his personal repository, and here he is the absolute master. The ultimate goal is simple - a set of changes, based on the master branch of the main repository, in any branch of the personal one.
At this stage, the biggest challenge is getting used to writing unit-tests for your changes. The most interesting thing is that the difficulty is not to write these tests, but to train yourself to do them. But if you develop such a habit, it will pay for almost any effort on the writing of tests. Start small: cover positive scenarios. They are easy to think through, easy to implement in tests. Having got used not to be afraid for breakage of the declared functionality, it is easy to add negative scenarios. This will eliminate the fear of irreparably breaking in unexpected places. Are the changes ready for the demonstration?
Changes are watched by others
On the set of possible cookies and implementation options, this is the most technologically advanced stage. The stage is not just called “watching others” - without the word “developers”. Automation!
To get started, the developer needs to start the process - to offer his changes to the merge into the main branch of the project. I think it is not necessary to say that telepaths are usually away, so the messages for each change should clearly reflect their essence. This is especially felt when you come back after a break to work on a project and discover other people's changes. Well, if it is clear what happened on the headings in the history of GIT. After all, after a vacation, I don’t really want, looking at the difference, to digest tons of changes to make a small correction. In open source projects, this is more than important - here hundreds of people are already watching your story!
This is where the CI server comes in. We use two: TeamCity & Jenkins . What exactly to use, we have not yet decided. It depends mainly on the preferences setting up a new build task. In order for the assembly to begin, as soon as the merge proposal arrives in the main repository, you will need the same glue — one of the extensions of the Pull Request Builders family.
For example, for Jenkins + GitHub, this is GitHub PRBP .
For Jenkins +, Bitbucket is a Bitbucket PRBP .
For Teamcity + Github, this is TeamCity.GitHub .
The CI server will carefully collect the proposed change, run the tests and be sure to report problems in the merge proposal itself:
Right here you need to configure and static code analysis. Now you can automatically find a huge number of errors in the code! For example, using the SonarQube . He will be able to measure the test coverage and warn you if it suddenly drops. Getting feedback takes minutes! You can correct errors in the same branch from which there was a sentence:
Do not be afraid to completely overwrite the story in this thread. GitHub, for example, will quickly update the sentence, and the CI server will re-compile it again. In many open source projects, such a rule is separately stated.
After successful assembly - time to inspect the code a living person. Of course, if you are the only developer, you may have no one to ask to see your code. In this case, it is an excellent chance to take a critical look at the changes made. After the assembly has been completed and the code is scanned, it remains only to press one button - upload the changes to the main branch. Responsible step. All systems reflect who did it.
With small changes, reading this section takes much more time than what it is written about.
Changes fall into the main branch of the version control system.
As soon as the changes hit the main branch, the time comes for integration testing. Here it is most convenient to put the task of assembling test packets on guard. We at our moment are updating the SNAPSHOT version in the Maven repository in our libraries. There are already chasing heavy integration tests that test interaction with third-party components, browser tests are run, where it is required. At this stage, it is quite possible to give the packages in manual testing.
What is pleasant and convenient this whole process? He is very flexible. All these actions are easy to spread into several cascades. For example, you should first make a merge request to an unstable branch, carry out all the heavy testing activities in it, and after them send a merge request to the main branch, within which you can assemble a package for sending to combat servers. And accept the request to merge only after a successful calculation. Or vice versa - all-all testing is carried out within one request, working and reassembling packages on the go, sending changes already ready for display to the main branch. It depends on the specific situation and ingrained habits of the team.
As soon as a sufficient number of changes have accumulated in the main branch, with which all participants are satisfied to the extent necessary, the time for release comes. Usually this is another task for the assembly with a manual call. In GitHub, for example, use the command for HuBot for this. It is very convenient to mark the head change with a tag at this moment. This will allow at any time if you want to be able to build the desired version without remembering the change number.
Choosing a release version is a separate process for which even specific standards exist. For small libraries, we use two-digit notation: the youngest grows each release, the older one, with the loss of backward compatibility, resets the younger one. For larger libraries - three-digit with the same logic.
I want it too!
In order to organize such a process, you do not even need to have your own CI server. For small projects it is very convenient to use www.cloudbees.com/jenkins , which will give a full-fledged Jenkins server with a limit on the number of assemblies. Starting to build merge offers on GitHub is quite realistic using buildhive.cloudbees.com - it has no limits. Or travis-ci.org , for which you need to put a small configuration file in the root of the project.
If you have your server at hand, set up Jenkins or TeamCity on it - it's half an hour. The default settings in these servers cover 80% of basic needs, and you just need to add the necessary extensions.
And having set up a complete chain on a small project once - it is possible to scale a solution to neighboring projects of even larger sizes in a couple of hours from absolute zero.
We will teach!
If you are interested in doing such things, it is interesting to build automatically scalable systems that facilitate development, assembly, and most importantly, testing, if it is interesting to improve the quality of products used from small to large, welcome to our School of Development Automation !
Source: https://habr.com/ru/post/237017/
All Articles