Engineering Assessment: how to measure the technical condition of the project?

You can improve what you can measure. This is the main thesis of measurement. We measure to improve. We want to improve the code, the engineer. For this you need to measure the code. How?

Today I will talk about metrics at the lowest level of creating IT products. About those metrics that are at the level of engineers, at the level of programmers and QA. Emphasis is placed on those metrics that depend on the human factor, which are not measured by automatic tools. Working on several projects and observing dozens of others as Agile-trainers, we developed 9 metrics that describe the current state of the system from an engineering point of view. In dynamics, they help to react instantly if something goes wrong.
')
If the topic is interested, welcome under cat.

Business, Product and Project Metrics

First, a little about metrics at higher levels.

• We measure business performance by company value, value of shares, market shares, growth rates. These indicators are clear and have long been settled. Rising stock prices - more income from shareholders.
  Going down below, a business makes a product that brings some value to users.
• The product has its own metrics to work with it: the cost of attracting a customer, retention, the number of paid customers who have unsubscribed, etc. These metrics are used by product developers for product development and prioritization. A business uses these metrics to predict its performance — profits, share prices, etc.
• Projects have their own metrics. Projects of a mobile application, website, etc. can be part of a single product. Project metrics help managers manage a project and respond to abnormalities. Speed ​​metrics, capacity, lead time, if you use Kanban. These metrics also help product managers and business owners predict development. Communication is again traced to a higher level.
• Going down below, we get to the code level, and this is where the problems begin. There are no metrics that help project managers build forecasts regarding project development. The metrics of complexity, the number of duplicates, test coverage do not help to find a connection with product metrics and business. We will introduce you to what metrics will help to associate the code with the goals of the product and business.

Code Metrics

Having worked on several projects, both small for 2-3 months and long-term for 2-5 years, examining dozens of projects since 2008, we concluded that the teams use similar tools to assess the status of the code.

We formulated the accumulated experience in the form of the Engineering Assessment Framework, which assesses the product in terms of engineering quality and helps to focus on metrics that help business development.

The Engineering Assessment Framework consists of 9 questions that can be answered in 10 minutes. Some of the questions are tied to the human factor, therefore, it will not work to collect the answers from the automatic system. Over time, we will adapt and improve issues, expand metrics to better describe the situation, not forgetting backward compatibility for those who are already using the Engineering Assessment Framework

In more detail, we’ll dwell on what these 9 questions are, how to approach them and what the priority metrics map is.

1. How many people in a team can explain each particular piece of code?
   Measures cross-functional. Most teams understand the value so that anyone can work on any piece of the application. Bus-factor, illnesses, long-planned vacations, distractions to third-party rallies, meetings or support for old projects that the manager throws off. Lack of cross-functionality leads to costs. Even if you have a stable speed of 10 points per week, there are no guarantees that you will make 10 points next week. Because the only specialist in the current field, on which the team works, was unwell.
2. How many manual steps are needed to enter Production?
   Direct reference to automation. Automation should be on all fronts to avoid human errors, as well as speed up the deployment process. The speed of delivering value to the user comes to the fore.
   
3. How quickly did the team find out that the fresh build was broken?
   The speed of receiving feedback is an essential element of development. Back to the level of automation. You can learn about a non-working build in 10 minutes if the build crashes. And it is possible in 2-3 days, when you ask to check her tester, who is now busy, tomorrow he will have a day off, the day after tomorrow he will come and test it. After all, during this time you will already make changes to the code.
4. How long in the last week / iteration has the fresh build remained inoperative?
   A dead build means we cannot deliver value to the user. All subsequent changes are meaningless, because they are made on the basis of incorrect conclusions. If a team stops working and instantly rushes to fix a non-working assembly, this is a good sign.
5. Technical debt increased or decreased over the last week / iteration?
   One of the best questions. Often you had to hear this: “Well, here we did it, it would be necessary to refactor, I wrote ToDo, then we will do it.”? Or "Everyone is ready, the tests just did not write, but now there is no time for them, then we will write." Such "then" can grow like a snowball, driving the project into a debt trap. Technical debt will have to pay. And the higher it is now, the higher will be the interest on it.
6. How many changes to the project did the team send in testing in 1 hour?
   No typos. It is in one hour. Not a day, not a week, but an hour. What is change? Moving the button - change or not? Change - any modification of the system that brings the user some value. Values ​​must be related to business objectives. This question well shows how the team is able to decompose tasks. And not just for technical tasks, but for those that bring value to users and businesses.
7. How many changes have you rolled out to Production in the last week / iteration?
   Another reference to automation and decomposition, but more focused on facilitating the exit to Production. When answering this question, think about how many bureaucratic steps do you use? How many reviews before exiting, any additional frauds that can be avoided? If the procedure for access to Production is devoid of this husk, you will be able to deliver value to users faster and more often.
8. How many people in the team sent the code to the central repository at least once a day for a week / iteration?
   Feature-branch is used frequently. This is suitable when each individual functional part is done in its own branch and will be sent to the central repository sometime. When? And who knows? Typically, this approach leads to the fact that the changes do not flow into the central repository for weeks or even months. The longer you marinate changes in your own branch, the harder it is for you to integrate into the master. We are in our practice faced with companies in which the integration of changes takes 1-2 months. Month, I'm not kidding!
   
9. How quickly do you get feedback from the tests after writing a single line of code?
   Every time developers are doing something new, they need feedback on what they are doing. Fidbek from automatic tests you will receive after 5 seconds. Feedback from the tester - in hours or even days. The faster you get feedback, the faster you will correct your actions. It is like a GPS. Imagine if you turned the wrong way, and he would only inform you about it in a couple of hours.

These 9 metrics enable project managers and product managers to predict when this or that functionality will be done, when we will connect new customers, when the business will hold a larger market share, and when, finally, shareholders and the team will get more money.

Priorities of metrics

To determine at what point in time which metric is worth improving, you need to understand the goals of the business and what stage it is now. At the stage of launching a new product, when every day of delay can cost a lot of money, loss of market share or lost moment, you can, for example, ignore metric 1, which is responsible for cross-functionality. At the same time, at a fast pace, automation is crucial, and you need to pay as much attention as possible to those metrics that show the degree of automation when deployed to Production. Technical debt is also not critical at the very beginning, it grows there.

When a product enters a quieter stage, the importance of a cross-functional team increases. It is also important to remember that the constant growth of technical debt will drive the team into the pit and it will become unable to develop the product. It turns out that even for one product, at different stages of development, the priorities of metrics will be different.

This leads to the creation of a certain map of metrics that takes into account the interests of a particular business and its plans for the near future. We have developed a common map of metrics, which will work 90% of cases and will serve as the initial mark for work on metrics. In the process of product development, priorities evolve.



A green field in a metrics map means that it is important at this stage. Red - the priority of this metric at this stage is lower than the others. Such cards will be compiled individually for each product and business.

findings

The result is a set of metrics, which we call the Engineering Assessment Framework, showing:

• The degree of cross-functionality of the team
• Automation level
• Focusing on delivering value to the user (here is a reference to how well the team decomposes the tasks, while preserving the value for each user and business)
• Code quality (technical debt is about that)

What's next?

Now we are recruiting teams who want to try an approach to measuring the technical condition of a project and will work closely with them. First of all, in order to help them improve their projects, as well as, to adapt metrics for specific cases (we also do not exclude this). It is possible that over time, the metrics will adapt to the changes and will be settled after some time. If you are ready to connect to such an experiment, write.