We haven't talked about development automation for a long time. Therefore, this time I will talk about how we have reduced the release time from three days to one and removed from the process the participation of a person with his possible mistakes.

To talk about the long and thorny path is always difficult. However, in recent years, the Yandex.Money development infrastructure has made a big step towards automating the most important process for us, the release, which is simply not a sin to tell. In fact, we got a complete solution of Continuous Integration and Continuous Delivery based on the Bitbucket , Jenkins and Jira bundles.

Who needs all this

Continuous Integration and Continuous Delivery (CI / CD) processes have been used for a relatively long time in the development of Yandex.Money, which allows delivering changes to the production in small portions and maintaining a high release rate.

In general, a complex process for releases is not necessary for everyone and not always. For example, it is hardly useful for start-ups because of the many unnecessary stages - it is easier for them to get together and discuss everything in a narrow circle. But if it doesn’t work out so well in your structure, and an insufficiently fast release process threatens with a dozen critical changes in production at a time — you should think about automating the development.

In addition, an automated process can significantly reduce the workload of teams. Without CI / CD automation, you risk becoming bogged down in human error and a lot of manual operations. For example, we use the popular Jira task tracker, which is perfectly suited for task management and integration with the Bitbucket repository, but it is difficult to automate the release cycle with it. Therefore, our development process before Jenkins looked like this:

Release process Yandex.Money.

Development tasks and bug fixes were performed by the Successful Git Branching Model , and testing took place at the Feature Branch. The whole hell of manual operations took place just as the Release Candidate was being prepared for acceptance tests. The developer needed to complete the following steps:

1. Cut the release branch from dev to prepare the next release;

   
   

2. Collect future release code;

   
   

3. According to the comments to the commits, find all the tasks included in the release of Jira and make connections for them with the release in Jira;

   
   

4. Create tickets to Jira for load and acceptance testing;

   
   
5. Finally, the release manager needed to check that all the tasks associated with the ticket were tested and did not appear after or during the tests.

Working out these five points and coordinating the release with the responsible persons took us from 1 day to a week. It could not go on forever, so the automation of this process on Jenkins soon appeared.

What we will automate

Four years ago, we realized that it was time to change something: the size of the system, the number of employees and changes on the combat servers were growing before our eyes. Since 2014, the number of releases has increased 3 times - from 45 to 150 pieces per month.

At that moment we ran with the following baggage:

• The payment service consisted of several monoliths;

  
  

• There were no mandatory autotests;

  
  
• There was no automated code review and application deployment.

After the release was approved for deployment in production, a team of operating engineers took up the case. Since the components of the payment service work in cluster configurations for us, the update deployment for each component resulted in a dozen iterations. Only the deployment of releases on several environments daily engaged 5 people.

With all this, you can live when there are a few releases and the staff is such that you can get together in a negotiation room and quickly discuss and coordinate everything. But at some point it became a serious problem and a brake in the increasing pace of releases. We have begun a long journey of rethinking the development process, which can be briefly summarized as follows:

In fact, even a fully automated release preparation process will not significantly increase the pace, since we are pushing into the performance of the operations department - at one point in time at production we have decided to deploy only one update. This is done to make it easier to diagnose possible errors due to a release.

It turns out that you need to automate not only the formation and testing of the release for the transfer into operation, but also the deployment itself. That is, to improve the processes of development and operation. The first attempt to automate the operation is described in one of the previous articles - the experience was useful, and the mistakes made allowed to bring to the technical solution. Now it is being implemented for most components and in the future, most likely, will be the reason for another publication.

Why Jenkins

We used to use the JetBrains Teamcity code to build, while the growth in the number of developers and services has not shown the economic inexpediency of such a solution for Yandex.Money. Therefore, they turned to OpenSource and, as a “second version,” made something comparable based on open and free Jenkins .

Tasks under automation turned out the following:

1. Dynamic generation of Jenkins tasks. Allows you to create and maintain up to date tasks at the component level and the branches inside them;

   
   

2. Preparation for testing :

   
   

   1. Cutting the release branch. A button in the Jenkins panel in Bitbucket creates a new release branch from the dev branch, to which all changes from developers have already been added;

      
      

   2. Build for testing. A separate script builds the package for testing. Jenkins creates an RL ticket in Jira and binds to it the descriptive tasks for developers;

      
      
   3. The script generates tasks for acceptance and integration testing. Jenkins sends an email with release information to all observers of the released tasks;
   
   

3. Production Deployment :

   
   

   1. Build for production deployment. At the team of the developer, Jenkins adds the release branch code to master, and from master to dev (for the developers to have the current version), then builds a package for deployment;

      
      
   2. After assembling, the Jenkins script creates a task for Jira in deploying a release for the operations department and sends it to the person in charge of the post office - you can upload it to production.
   
   

Let us examine in more detail each of the stages.

Dynamic Jenkins Task Generation

When adding a new file with a description of the task to the repository of the component, Jenkins will automatically create an assembly task for it. This is the responsibility of our SyncBitbucket mechanism, a working example of which lies on Github .

Interaction with Bitbucket occurs through the API, and with Jenkins - using the Job DSL plugin.

The steps for dynamic generation are as follows:

1. Manually create a Bitbucket synchronization task (file synchronizeBitbucket.groovy ). The task goes over our Bitbucket projects and generates tasks for synchronizing individual projects. A project is a set of repositories in which we have entities similar in purpose: services, libraries, and grad-plugins. The launch of this task takes place on schedule, because we cannot track the appearance of new projects.

   
   

2. The project synchronization task (the synchronizeProject.groovy file) is also launched according to a schedule, searches for repositories and creates a task to synchronize each of them.

   
   
3. The task of synchronizing the repository (file synchronizeRepo.groovy ) is the main part of our mechanism. Its purpose is to bypass all the branches, find in them the script files for generating the task and launch them for execution. The task is launched by committing to the repository.

The scripts for generating tasks are written in the format of JobDSLplugin . In order for a developer to add a new type of task to Jenkins, it is enough to place the file with the prefix jenkins_ in the \ Jenkins folder of the root directory of the repository. In the task creation script, you can use the data supplied by the repository synchronization mechanism, for example: the name of the branch, the address of the repository, and the address of the directory in Jenkins.

Example of the description of the task for cutting the release brunch:

folder(jobParams.jenkinsDirectory) if (jobParams.gitBranch != "dev") { return } job(jobParams.jenkinsDirectory + "/createReleaseBranch") { jdk('jdk1.8.0') scm { git { remote { url(jobParams.gitRepoUrl) refspec('refs/remotes/origin/*') branches("**/$jobParams.gitBranch") } } } steps { gradle { makeExecutable(true) description('  ') tasks(':createReleaseBranch') } } } 

Such a mechanism allowed introducing the stages of the release cycle with minimal effort and for each component separately. It turns out an analogue of Travis CI , and you can create your own set of assemblies for each branch.

Preparation for testing

To automate the Jenkins release cycle, it was necessary to create several scripts that look like “launch buttons” for each stage of the release cycle:

• createReleaseBranch - cutting the release branch;

  
  
• prepareReleaseForTesting - build a test release, setting a task for testers.

Scripts are created just at the stage of dynamic generation of tasks for each component. That is, the start of one or another stage of the release cycle means the launch of a specific script of the component being prepared for release.

After launching createReleaseBranch , a release branch is formed from the dev branch in Bitbucket, and a separate folder is created in Jenkins. The name for the folder is automatically selected according to the “release_version” scheme.

Next, the programmer needs to go to a new folder and run a separate prepareReleaseForTesting script there , which will assemble the update package and send out release build notifications to all interested. In addition, Jira creates tasks for acceptance and load testing.

Immediately after sending a new release for testing, you need to tell all interested parties about this. So the product manager will be able to agree in advance with marketing and PR about the public announcement, the leaders of the directions will be aware of what is happening, and technical support will be able to work out the answers to possible questions from users.

For all this to happen, it was important to make Jenkins friends with Jira. Here the whole question rested on the organizational aspects, since such developers are often encountered when changing code in Bitbucket:

• "The button sometimes got out of the screen - corrected";

  
  

• "Added a strip on top";

  
  
• "Corrected checking for numbers, should no longer swear."

With such descriptions, it is difficult to understand to which task in Jira are code changes. Therefore, it was agreed with the developers that, first of all, a comment on changes in Bitbucket will contain the task number (for example, “BACKEND-186 Fixed sending email”). This number was before, as the developer needs it to analyze history in Git and Bitbucket, but now it has become critical.

If the human factor comes into force and someone forgets about it - Jenkins will add a comment “as is” to the general release letter, and then the project manager will bind the task to RL in Jira.

Deployment

During the deployment process are the following two tasks in Jenkins:

• mergeToMaster - merging the tested code in the release branch with master;

  
  
• prepareReleaseForProduction - build the release from master and set the deployment task for the operation team.

After testing, the person responsible goes to the release folder in Jenkins and runs the mergeToMaster script, which executes the following in Bitbucket: merge release-> master and master-> dev.

It remains the final stage - assembly for production. This is the responsibility of the prepareReleaseForProduction script, which assembles the .DEB package, sets the task in Jira to the operations department and notifies the person in charge that the release is ready and can be uploaded.

As a development project for the automation of the release cycle, we are now building a link with integration tests in the testing department, in order to make the testing stage fully automatic in the future. In order for developers to release more often, we taught Jenkins once a day to automatically send notifications about new tasks in the dev branches. In addition, it is hoped that this will prepare developers, testers and system administrators for future automatic release clipping.

It is curious to know about your approach to releases - how do you struggle with human errors, do you support any stable release rate?