Daily releases are not so scary

My name is Oksana Kharchuk, I work as a QA-engineer in DataArt for a little over a year. I'll tell you how the work process is organized in our project, and how to be if the release is every day.

At first, when I first came to DataArt, I associated the word “release” with something scary. But, as it turned out, if the work process is built correctly, releases even every day are not at all scary, but very convenient. To achieve this, the development process in our project is built on the principles of continuous delivery and continuous integration. .
')

What is Continuous Delivery and Continuous Integration?

Continuous delivery or continuous software delivery - a set of practices and principles aimed at building, testing and delivering software faster and more often. Continuous delivery of quality code relies, in turn, on continuous integration.

Continuous integration, or continuous integration, is the practice of software development, which is to perform frequent automated project builds to quickly identify and solve integration problems. After all, it is clear: if several programmers work on different parts of the code, many difficulties arise when integrating these parts. Continuous integration helps to cope with them.

How does continuous integration work?



Developers download the source code from the repository, make any changes, and then send their changes back to the repository. After that, the project builds automatically, tests are run and a report is sent that everything in the modified code is good, and it is saved.

And here it is shown in detail what happens at the testing stage after saving the modified code:



The first thing that runs is jar files, then in parallel :

• war files;
• external services,
• integration tests;
• JS tests;
• static code analysis;
• tests with a database.

Then, one after another , the following actions are performed: validation build, build installation, deployment on a test environment and, finally, Selenium tests.

I want to make a reservation that this is not happening in every project - this is just what the developers decided to do for themselves, to be sure that their changes did not break anything.

This is how the process of continuous delivery is built. It is also worth saying that the practice of continuous delivery should not be confused with the practice of continuous deployment. The difference is that with continuous delivery, manual testing is present at a certain stage, and continuous deployment is performed fully automatically:



Here I am talking about the continuous delivery - further I will explain the essence of manual intervention in testing.

Flexible development methodology

In the project we use a flexible development methodology - Agile. Flexible methodology involves the development of software cycles. Each cycle represents a smaller version of the IT project: requirements analysis, planning, development, assembly, testing, deployment. At the end of the iteration, the customer receives a ready-made version of the IT system, and, if necessary, further project priorities are revised. Then the development cycle starts again. The result is a solution that meets the requirements of the customer.



Although releases occur every day, this, of course, does not mean that we add new functionality every day. When the development of new functionality occurs, we go through all the above mentioned stages. At the same time, we have no time limit: if we develop a new functionality, it will not necessarily be done for two days, two weeks or a month - how much time is postponed for this project, so much time and the development process can last, and we need to take into account that does not immediately go to the finished version of the product.

However, the following question arises: if the code is stored in the repository permanently and if we have continuous integration, how to hide the code that will implement the new not yet ready functionality? This is done using the configuration flag, with which the developers hide the code that is responsible for the unprepared functionality (although, of course, there are also changes that cannot be hidden under the configuration flag).

The process of developing a new functional:

So, how do we develop new features? First we get the requirements, after which the QA-team, having analyzed and clarified the requirements, begins to write test cases. At the same time , the development process is underway. After the developers, having given the go-ahead for testing, turn on the configuration flag on the test environment, QA-engineers begin functional testing. While QA-engineers are testing and finding errors that developers fix, automators or the Selenium-team (as we call them), begin the process of automating test cases. After functional testing, regression testing is performed.

Release process

If all is well, the decision is made that the new functionality will go into production. How it's done? It's very simple: the developer commits the inclusion of the configuration flag, under which the new functionality was hidden. After the commit, a new version of the product is deployed on a test environment with new functionality already available. The developer makes the inclusion of the configuration flag just before the day when the release is assigned, so that the change is included in the new version of the application, the one that will be released to the production.

So, after the flag is turned on, and our functionality is available on the test environment, acceptance tests are run. The process of acceptance testing runs every night, regardless of whether we release new features or fixed bugs. The test suite is run overnight twice to minimize random drops. These acceptance tests cover all high-priority cases. When the QA-team arrives in the morning, it looks at the night testing reports, and it happens that everything did not go smoothly: 1 - 2% of tests dropped or were not passed. In order to make sure that these were just random drops or, conversely, that automated test cases revealed an error, the QA-team will manually go through these test cases.

Then, if all is well, we report it to the customer, and we can release changes to the finished version of the product.

After customer approval, our application is deployed on an intermediate environment, which is very similar to Prodovskiy, and acceptance testing is carried out here. It is carried out by the QA-team, as well as by Selenium-tests - manual and automated testing takes place in parallel. If everything went well, we inform the customer about it and release our version in production, where smoke testing is conducted. This is the perfect scenario.

What if there is a significant bug in the morning? In this case, the release process stops until the error is corrected. If it is possible to correct the error quickly (until the end of the day, for example), a new tag is cut off in this branch, and the same branch, but with a new tag, is released to the intermediate environment, and the release process is repeated again.

Every night if the release is successful, the branch is cut off from the branch, i.e., our application branch, which is sent to production. Until the cut is released, the new branch will not be released. Of course, it happens that we do not have a release for three days - when a bug interferes or there are failures in the process of cutting a branch.

Minor bugs are fixed for the next release. This is the advantage of daily releases: if today you find a small mistake, you will quickly fix it, and tomorrow it will not be.

In general, the release process is as follows:

• View a report on the passage of automatic acceptance tests on the test environment.
• Run fallen and missed tests manually.
• Acceptance testing in the intermediate environment, which is most similar to Prodovskim.
• Smoke testing prodovskom environment.

Please note that the new functionality is not released on production every day, but this does not mean that there will be no release. Will be! The machine is running, the release process is debugged and happens every day. If there is no new functionality, there are always fixes of existing bugs.

Role and Responsibilities of a QA Engineer

You might get the impression that everything is so automated in us that the QA engineer has nothing more to do. But in fact, work is enough.

• First, the clarification and analysis requirements. This is done by QA-engineers - after all, when a project manager makes demands, a lot of things are often missed.
• Secondly, the assessment of the time spent on working with the project. When a project manager asks how much time is needed to write test cases, functional testing, etc., you need to calculate all this.
• Thirdly, work with test documentation, writing test cases, compiling checklists and report bugs.
• Fourthly, functional, regression and cross-browser testing is carried out in our project (since we have a web application, and each browser has its own mistakes).
• Fifthly, QA-engineers are engaged in the support of the release: every morning the passing of the fallen night tests, as well as acceptance and smoke testing. But this does not mean that the whole team is busy release. The release is occupied by people who have less priority tasks - our team is self-organizing.

There is also a release coordinator - the person who is responsible for the release. He prepares an error report for the customer and conducts negotiations with him. From the customer’s side there is also a special person who communicates with our team.

It is also worth telling what exactly the automators are doing in our project.

Role and responsibilities of the automator:

• Development of automated tests.
• Support existing automated tests so that they are relevant.
• Release support. From the Selenium-team we also have a release coordinator who is directly responsible for the night tests and their support. Thus, he is responsible for the release and can also communicate directly with the customer.
• Support regression testing.

Principles of continuous delivery

Finally, I would like to talk about the main principles on which, in my opinion, the practice of continuous supply is based.



1. The process of release and deployment should be repeatable and reliable. Once a process is running, it can no longer be stopped - even if you do not release anything new.
2. Automated tests should cover 80 - 90% of the functional - without this process will not work.
3. It is necessary to take frequent tests on vulnerabilities. If you see that in some place you often have tests or a lot of mistakes, you need to test this place more often and make sure that improvements occur and something is being corrected.
4. Quality needs to be built right away. This applies not only to developers, but also to testers. It happens that QA-engineers check the bug reports on the local developers' machines in order to prevent non-fixed bugs from entering the common repository. Then the developer fixes the error locally, it is much cheaper and faster.
5. Everyone is responsible for the release. Because of this, we have a well-established communication. If QA-engineers have questions, we can calmly write to the developer, and the developer can also ask us about something.

Despite the advantages, the concept of continuous supply has, of course, disadvantages. This practice is not suitable for all projects - whether it will be possible to implement it depends on the project and on the team that deals with the project. And for me the main disadvantage is the routine of the process. Coming every day, we see the same application, we run tests and, naturally, we can miss something, because the vigilance is lost. Therefore, we in the team try to break everything apart, and one person doesn’t do one thing for a long time - on different days a person can perform different tasks. It brings variety to the process.

The main advantage of continuous delivery is a satisfied customer, because he can get what he wants very quickly and have an income from it.