Codefreeze and continuous delivery

Traffic jams start at intersections when traffic flows in conflict. While one stream is moving, the other has to wait. Something similar happens with the release of the next release of the software product: to release a product of decent quality, you have to stop development.



There are no programs without errors. By doing something useful, we unwittingly add to the code that which we did not intend. Sometimes surprises occur at the junction of seemingly unrelated changes. Here's a wonderful example of how optimizing a mouse driver makes it difficult to update your address book in Outlook . Therefore, there is always a time when the team focuses solely on testing and correcting errors. Adding new features at this time is prohibited.



But the sources of new requirements, wishes and other hoteles do not become impoverished. While the release is stabilizing, the queue at the next traffic light / product backlog is only growing. It turns out a classic cork. How to resolve this conflict?





Extreme programming suggests correcting errors as they are discovered, without stopping development. And in general, it is necessary to program without errors. I like the cool method, but unfortunately, developers still program with errors. I wonder why? It will be necessary to ask on occasion. In general, traffic jams in this case does not occur, but a certain number of accidents / annoying errors inevitably seeps into the release. If the cost of correcting the leaked error is small, then this approach is fully justified.

')



The other extreme is the classic Waterfall, which calls not to mix one with the other. First, design everything, then carefully implement, carefully test, correct errors and transfer the result to the customer.



The approach is quite reasonable in terms of cost optimization, but often not very effective in conditions of uncertainty. When creating new products it is often not possible to reliably find out what should be implemented and how. Much is done "on trial" and is redone later. With the successive passage through the phases from the idea to its verification, too much time passes.



Moving along the beaten path full of trial and error, I think we managed to find a reasonable compromise between these two extremes - to provide the flexibility needed to develop new products, while maintaining a high level of quality. In this article I would like to share our experience, maybe someone will find it useful.



At first we tried to work with short iterations, which resulted in a working, tested version. Limited iteration skoup helps the team to concentrate and not be distracted. To have time to eliminate all the roughness to the finish iteration, it is customary to lay some time to stabilize the code. At this time, major changes are not made to the code, only errors are corrected.



The stabilization period is necessary, it limits the number of changes in the code and makes the process converge. In practice, it turned out that a significant part of the team at this time is bored: there are not enough mistakes at all. Of course, there is always work outside the iteration, but no changes can be made - stabilization is underway. And at the beginning of the next iteration, testers miss the pending new errors ... The management, seeing that the workers are idle, threatens to transfer them to other projects.



The next step was an attempt to work on the Kanban method. Tasks are a continuous stream, after the completion of each task version is obtained. But the hope of quality destroys the basic law of organic chemistry: if you mix a barrel of honey and a spoonful of shit, you get a barrel of shit. The situation could have been saved by the full testing of the version after each task, but this turned out to be too expensive both in terms of effort and time.



It would seem that the situation is hopeless ... But who said that changes need to be made in the same place where stabilization is going?



Now we are working on the model of "three levels of instability":

1. The development branch is poorly controlled stability. Here tasks are made "for the future", which will not be included in the current release.
2. Release branch - controlled reduction of instability. The set of changes that will be implemented in this thread is limited and fairly rigidly fixed.
3. The stable version is a state fixed at some point with an acceptable level of quality. Changes are made extremely rare and very small (hotfixes)

It works like this.



From the product backlog, we recruit tasks for release in such a way as to be done in a month. This is our current sprint. The team starts to implement the release, and closer to its end, when there are few changes and there is not enough work for everyone, smooth preparatory work begins for the next sprint. Works are carried out in a separate branch in the gita, in a separate section of the kanban board in Djir. When the current release's scop has been completed, tested and stabilized, the release branch becomes stable, the development branch becomes the base for the next sprint, and the development branch stops waiting for the end of the next sprint.





Allow, someone will say, but then in the development branch there are no current edits. Yes, and with errors inconvenient, they have to be edited in two branches at once. To avoid such problems, we agreed to preface any work in the development branch with an infusion of changes from the release branch. After this procedure, the development branch contains all current developments and bug fixes.





Changes from the development branch do not fall into the release version until the end of stabilization. This makes it possible to complete the release and transfer the stable version to the sales department. And before the next release, all the developments fall into the release branch.





Now we have a continuous flow of tasks and a non-blocking release stabilization procedure that allows us to release high-quality versions without stopping development.



The next point of application of efforts is the reduction of the stabilization period to reduce the gap between the branches and reduce the total time of TimeToMarket.