Lannisters always pay their debts! (and technical too)

Technical debt management practices in a single team

About a year ago, our team moved from a phase of accelerated functionality to smoother development with a focus on quality improvement. At this point, a significant number of non-optimal solutions, ugly code, outdated libraries have accumulated in our products. With all this it was necessary to do something.

To date, we managed to build a process that makes the fight with technical debt predictable, painless and inevitable.

What was obtained as a result of:

• The team is pleased. In the release retrospect regularly appear positive points about improving technology and reducing technical debt.
• Several quarterly releases in a row, we were able to increase the functionality without increasing the number of lines of code in the project. Removing unnecessary code and simplifying the necessary reduced the size of the code base for the existing functionality. And this reduction just about coincided in scale with the new code that implements the new functionality.
• During refactorings and upgrades, the product is always in working condition. Every two weeks we release a fully working interim release.

Let's tell how we achieved it.

What is technical debt

My working definition of technical duty is the amount of work that needs to be done in order for the project to fit the team’s vision of the beautiful. Note that technical debt may arise not only due to the liberal use of crutches in the design, but also due to a change in ideas about the beautiful. For example, generally accepted practices in the industry have changed. Or the developers fell out of love with OOP and fell in love with functional programming. Or once a fashionable framework is no longer a cake and it has become difficult to find specialists who would like to write on it.

However, the main cause of technical debt is entropy in all its diversity. Disabled unit tests, outdated comments that have lost contact with the code, unsuccessful architectural solutions, implementation of features that no one else uses, a reserve for the future that has not come, and much, much more.

From this it follows that the emergence of technical debt is inevitable in any long-lived project.

When technical debt is not a problem

What is bad technical debt? It increases the cost of further development due to a number of factors:

• Reduced speed of implementation of new functionality.
• The probability of regressions increases with the correction of defects.
• The development process becomes less predictable, therefore, less manageable.
• The process of introducing a new developer into the project is extended.

These losses are sometimes called “technical debt interest”.

There are situations when these losses are cheaper than eliminating technical debt:

• The end of the project's life is near. Notice, not the end of adding functionality, but the moment when you can stop spending the efforts of programmers and support too. The same category includes disposable prototypes, disposable demo versions for exhibitions, etc.
• The value of development time is currently much higher than expected in the future. Urgent corrections to the fixed deadline, startups that run out of money from the next round of funding, etc. In such cases, the correction of technical debt can be postponed until the moment when the intensity of emotions decreases. There are projects that do not emerge from the state of trawling, but the advice from this article will not help them anyway.

What to do with technical debt? Unsuccessful approaches

Approach number 1. "We do not have time, the release must be passed yesterday"

The desire to be in tomorrow's deadline at any cost, to the detriment of the speed of development the day after tomorrow.

Sometimes a pre-fabricated crutches design is an objectively correct choice. In my practice, this was most pronounced when making demos for exhibitions. The date of the event is fixed, if he did not have time for an important exhibition - the next attempt will be in a year. At the same time, you can show the product “out of hand”, carefully avoiding all the bugs. I, as an engineer, make such projects unpleasant, but the crutches in them are justified.

When you make a product that will live a long time, everything is different. Catching up on time due to dubious technical solutions is expensive. The total cost is:

1. from the implementation of the crutch itself,
2. from its subsequent replacement with a complete solution,
3. from suffering from the presence of a crutch in the interval between the preceding paragraphs.

The second point is very easy to underestimate, but about the third, the most expensive, there is a risk not to think at all.

When you meet with a twitching eye at a technical conference, it may turn out that it was the nightmare of endless debugging of the crutches construction that brought him to such a state.

Approach number 2. "Yes, here we must throw everything out and write again"

The worse the situation with technical debt, the stronger the temptation to bury the entire project code and write everything anew. This is one of the classic mistakes that can kill the whole project.

The topic is so well disclosed in the well-known article by Joel Spolsky that I see no reason to give my arguments.

Things You Should Never Do Part I

Approach No. 3. “We will refactor at night and on weekends so that the manager does not recognize”

It is not always easy to argue for the need to eliminate technical debt with a projection on profit. The development team may be tempted to bypass sharp corners and start a major refactoring without prior order. This can be done during off-hours, in pauses between other tasks or “on the tail” of other tasks due to inflating assessments.

What's bad about it? Oh, a whole bunch of things:

• Reduced transparency undermines trust between management and the team. Often the following attempts to rectify the situation by imposing discipline and tightening the screws will lead to a further deterioration in teamwork.
• The entrenched situation in which the priorities of the team and the management contradict causes demotivation from both sides.
• The results of refactoring in an unpredictable way are included in the product and cause regression bugs where they are not expected. In conscious teams, this leads to a sudden and unplanned load on QA. The unconscious goes into the product line and breaks there already.

After the team uses this recipe, the eye begins to twitch at the management.

Our principles

1. Add technical tasks to the general backlog

There are a number of patterns of life tasks in projects:

• A task that is not in backlog is less likely to get into work.
• A task without clear wording is less likely to get into work.
• A task that cannot be assessed with a high degree of confidence is less likely to get into work.
• A big task will wait in line longer than a small one.

Maxim Dorofeev in his “Empty Inbox Technique” tells very well about these things .

So that technical debt is not accumulated, work on its elimination should be drawn up with an eye to the principles listed above.

All tasks, except for the smallest, are got in backlog. So they have a chance to be made not only in their free time, but also as part of the planned work. In addition, such tasks are harder to completely lose sight of - the backlog is viewed more often and more closely than the TODO in the code, pieces of paper on the monitor, abandoned wikis, tea-filled napkins with diagrams and other sources of information.

• If the code has a non-trivial TODO, it contains a link to the task in the backlog. We check compliance with this principle for code review and do not accept complex TODOs without links.
  Behind the commentary there may be a drama

  Once alongside such a TODO it was written: “A crutch. Product Owner made me do it. Clean as soon as possible. ”
• When a developer realizes that a place requires refactoring, he creates a task in the backlog.
• When the developer has a wish to improve the platform, he creates a task in the backlog.
• Long-term architectural plans are backed up in the form of individual tasks, as soon as there is enough certainty at least in the first steps.

As long as such changes remain inexpensive in terms of development costs and the amount of testing needed, we can gradually improve our code base without interrupting business tasks and introducing additional risks.

2. Plan technical stories based on business priorities.

If a tree fell in the forest, but no one heard that, was there a sound? If there is bad code in the project, but this module will never need to be changed, is there technical debt?

I think that when it is unpleasant for a programmer to look at some old module - this in itself is not a very big problem. Much worse is what happens when in this module you need to add new functionality or extend the old one. Compared to making changes to well-written code, such tasks are more often (and stronger) beyond the original rating and contain more bugs. Sometimes much more. To protect ourselves from problems of this kind, we try to plan refactorings so that they are done before writing new functionality in the same place.

If changes in functionality and refactoring seem small, you can do them together. Empirically selected task size for which such an approach would be optimal - 3 days of work by one developer and less. When it is clear that there is more work, it is divided into refactoring with the current behavior and the implementation of new functionality.

Thus, the procedure for eliminating technical debt is determined by the order of business tasks in the backlog.

The “reliance on business priorities” principle has another application. One of the typical problems that plague developers who are trying to write well is the difficulty in allocating time to optimize performance, improve maintainability, or other things that do not directly figure in the work plan. For these improvements, you can almost always find a business need. Who does not want the system to work faster, more stable, cheaper to maintain? All these advantages can be assessed and, on the basis of this assessment, we can put tasks for improvement into backlog, along with any others.

So if you want to optimize performance, and you have to edit another boring bug instead, you may simply not be able to explain the benefits of optimization in a language that the owner can understand.

3. Leave the code cleaner than it was before you

Almost any code, except written recently, is a little behind the current idea of ​​the beautiful in the field of style and architecture. When it is necessary to change the code within a certain task, it is considered good practice to make all the safe improvements that are possible in the affected area. What could it be?

• Reduction to the current style of design modules.
• Change internal names of variables to more understandable.
• Simplify implementation while maintaining behavior.
• Local refactorings that do not make major changes to other modules.

Such improvements are expected to make the code better, but they will not create a significant increase in the cost of development or testing.

Due to this principle, the quality of the code is gradually increasing in the background, even in places where no separate refactorings were planned. Moreover, the more often we work on a certain part of the system, the better the code of this part. A pleasant contrast with projects where the developer spends the most time in the parts with the worst code.

4. Whatever happens, the system should remain in working condition.

One of the basic principles of SCRUM says that at the end of each sprint the system should come to a stable state.

“By the end of the sprint, the increment should be ready, which implies its compliance with the criteria for readiness of scram teams and readiness for use. It should be ready to use, regardless of the decision of the owner of the product to release or postpone. "

–SCRUM Guide

Any work on the elimination of technical debt is done in compliance with this principle.

Large transformations are necessarily decomposed so that any single stage can be completed in one sprint. For example, we changed the build system in two stages ( Angular 1.x: crouching webpack, hidden grunt )

We work with VCS on principles close to classic gitflow . Development goes in feature-branches, testing there. As a rule, such a branch lives no longer than one two-week sprint. A branch that lives longer almost always leads to additional costs.

Our experience clearly confirms this pattern. Every time we couldn’t complete a big refactoring in two weeks, it was accompanied by pain and suffering. And the longer the task was and the longer the open branch lived, the slower the work went and the more problems there were.

The need to always be a few steps away from a stable release creates one of the most difficult and interesting engineering tasks - the search for the optimal decomposition of strategic plans. Large-scale changes can be decomposed into separate independent steps. It is advisable to start getting benefits as early as possible. The better this breakdown is done, the more likely it is to complete the job.

What does our process look like?

Once a release we make a detailed review of the technical backlog:

• We close irrelevant stories (lost relevance, made in the framework of something else, duplicates).
• We update the description where the vision of the question has changed.

When business stories appear on the horizon, technical analysis is done and all the technical stories that would help in implementation are linked to the business history.

In preparation for the planning of the sprint:

• We check the connection of technical and business stories.
• We tie to technical stories all related bugs that can be cheaply fixed in the same place.

How to form the technical part of backlog

Having taken the lead role in the team, I asked each developer and QA what improvements in the product they most want to make. Most of the wishes were due to technical improvements to the platform and refactoring. As further experience showed, all key technical problems of the product were included in this set of wishes. So this practice can be used to quickly form a technical backlog from scratch or get a general idea of ​​the state with technical debt in a new project for you.

The current backlog of technical tasks is due to the practices described above and does not require separate efforts or analysis. In addition, new ideas on technical improvement of the product are added to the backlog. This makes any team member who came up with this idea. The main thing at this stage is not to lose the idea. Refinement and determination of priority occur only later, in the course of work planning.

findings

• Technical debt is inevitable.
• For most projects, eliminating technical debt is a good investment.
• If the technical debt is not eliminated, the development speed will gradually tend to zero.
• The temptation to throw everything out and write again can kill or severely cripple your project.
• The benefits of eliminating technical debt should be formulated in the key benefits for business, otherwise there is a risk that the task of creating new functionality will always be considered more important.
• Tasks to eliminate technical debt should be managed. Such tasks are no different from other project tasks in terms of accounting, planning, prioritization.
• There are regular situations when it is possible to reduce technical debt and at the same time solve a business problem cheaper than doing it separately. These features must be used.
• Sophisticated and timely code-style agreements and review processes help slow the emergence of new technical debt.
• Short iterations are useful for refactorings as much as for developing new functionality.
• The team usually knows where the technical debt is in the project and how scary it is. It is worth using this knowledge when forming the idea of ​​the technical debt of the project.