Key QA Process Indicators
Within Quality Assurance, various metrics and indicators of product quality and development process can and should be used. Metrics can be divided into groups according to the parameters on the basis of which they are calculated, according to the stages of the development life cycle, in which they are applied, according to the goals and objectives, according to the stakeholders for which they are intended. This list goes on and on.
In this article I decided to gather together and consider the most basic, in my opinion, groups of criteria and measures for the QA process. And in each group I will list only the most important and illustrative, again, in my opinion, metrics as well as analyze what they are necessary for, in what situations are useful and how to use them.
By itself, a metric in the context of software is a numerical expression of a property, the quality of the product itself or the process of its development. In other words, this is what we can use to measure, compare and evaluate software.
')
Now literally a couple of comments about the values ​​and properties of metrics:
It is theoretically possible to invent your own characteristic, formula or indicator for practically every, even the most insignificant action, stage or status of the QA process. You can take into account each artifact, all transitions of defects by status, calculate the number of tests in the set. However, the most important question that you should immediately ask yourself is when there is a desire to measure something: “Why is this information needed, how can it be used?”. When forming a set of metrics, one should make a start from goals, plans for improving processes and product.
So, in this article we will not consider the usual quantitative indicators of the progress of testing, which are used in most reports and statuses. Instead, we will analyze which areas, artifacts and areas of development from the point of view of Quality Assurance, we must measure and control in order to assess the quality of work performed. Analysis and optimization of these points are extremely important for effective interaction with stakeholders ( https://doitsmartly.ru/all-articles/sw-testing/120-stakeholders-for-qa.html ) and software development in general:
1. Requirements for the software being developed.
Absolutely, we must understand that we are developing and testing, and the degree of this understanding must be able to be assessed. Potential risks or missed problems at the specification level can lead to the most serious and expensive errors.
2. The quality of the developed product.
Everything is obvious: you must be able to assess the quality of development and software in order to make predictions and assess risks. It is important to understand how high-quality and reliable the product is, relying not only on the presence or absence of errors found, but just predicting whether there are many potential problems.
3. Capabilities of the QA team.
It is also simple here: in order to manage the testing process, plan work and predict deadlines, it is always necessary to have not only the actual status of the tasks, but also the capabilities of the QA team.
4. The quality of the testing team.
In addition to the quality of the product itself, it is necessary to measure the effectiveness of the QA process itself and the testing team. To constantly optimize and improve the quality of work, you need to know where we are now, what allows us to move forward and what is thrown back.
5. Feedback and product satisfaction.
The latter is an area, but, of course, not in importance, but in the opinion of the stakeholders of the process, consumers of our services, users of the product. It is very important to be able to measure the overall degree of satisfaction with the product, highlight trends and draw appropriate conclusions. Properly selected metrics for this group will allow you to identify possible problems in time and promptly apply feedback to improve processes.
Next, we consider exactly which metrics are included in each group, analyze how exactly they can be evaluated. For each group, I will give a few examples of possible metrics and describe their meaning. These and some other QA process indicators are discussed in more detail in my article “The Most Important QA Metrics” ( https://doitsmartly.ru/all-articles/sw-testing/133-the-most-important-metrics-in-qa. html ).
This group of metrics will allow you to assess how much we have worked out the requirements (user story) for software, identify vulnerabilities and the most complex, potentially problematic software features, and understand where special control is required.
1. Test coverage requirements
"Total number of tests" / "Total number of requirements"
The purpose of the metric: to identify weaknesses in the test coverage, highlight the risks.
2. The degree of interconnectedness of requirements
AVERAGE (“Number of requirements related to requirement No. 1” / “Total number of requirements -1”, ..., “Number of requirements related to requirement No.n” / “Total number of requirements -1”)
The metric is calculated as the number of links of each requirement with the remaining requirements. It uses the average value for all requirements.
Purpose of the metric: to provide a basis for assessing the timing of testing and taking into account possible risks. Knowing the degree of mutual influence of requirements on each other, you can, for example, plan additional time and cases for end-to-end testing, work out regression tests, look towards integration, etc.
3. The coefficient of stability requirements
“Number of changes in existing requirements” / “Total number of requirements implemented per iteration, including new ones”
Purpose of the metric: to show how many requirements already implemented have to be redone from release to release when developing new features. Also, the metric gives an idea of ​​how easily the system's functionality is scaled, new features are added.
This group of metrics allows you to evaluate and compare from release to release both the quality of the software and the quality of the development itself.
1. The density of defects
“Number of defects in a separate module” / “Total number of defects in software”
It is calculated as the proportion of defects in their total number falling on a separate module within an iteration or release.
Purpose of the metric: highlight which part of the software is the most problematic. This information will help in assessing and planning work with this module, as well as in risk analysis.
2. Regression coefficient
"The number of defects in the old functionality" / "The total number of defects, including the new functionality"
The purpose of the metric is: to show what the team’s efforts are going to do - whether it is engaged in more development and debugging of new features or spends most of the time on fixes in already existing parts of the software.
3. The ratio of re-open defects
"The number of re-detected defects" / "The total number of errors, including previously corrected and new ones"
The purpose of the metric: to assess the quality of the development and correction of defects, as well as the complexity of the product or a separate module.
The main task of this group of metrics is to express in numbers what the testing team is capable of. These indicators can and should be calculated and compared on a regular basis, analyzing trends, observing how certain changes influence the work of a team.
1. Speed ​​of operation of the QA command
"The number of story points for N iterations)" / "N"
It is calculated as the ratio of realized story points \ requirements \ user stories for several iterations \ sprints to the number of selected iterations.
Purpose of the metric: to express numerically the possibilities, the speed of the team’s work for the further planning of the scope of work and analysis of development trends. The metric allows you to monitor the speed of QA, to monitor which internal or external influences on the team affect this speed.
2. The average lifetime of the defect
"The total time to correct defects found" / "Number of defects"
The total time during which the defects found during the iteration or release were open to the sum of the defects.
Purpose of the metric: show how much time on average it takes to work with one defect: its registration, correction and reproduction. This indicator will allow us to estimate the time required for testing, to highlight areas of software with which the greatest difficulties arise.
The task of this set of metrics is to assess how well testers perform their tasks, determine the level of competence and maturity of the QA team. With such a set of indicators, you can compare a team with it at different time intervals or with other, external testing groups.
1. The effectiveness of tests and test kits
"The number of errors found" / "The number of cases in the test set"
Purpose of the metric: to show how many errors on average our cases can detect. This metric reflects the quality of the test design and helps to monitor the trend of its change.
2. The ratio of errors missed on the product
"The number of errors detected after the release of the release" / "The total number of errors detected before and after the release"
Purpose of the metric: to demonstrate the quality of testing and the efficiency of error detection - what proportion of defects was filtered, and what percentage went to the productive level.
3. Real time QA team work
"Time spent on target QA activity" / "Total working hours of the team"
The ratio of time spent by the team directly on target QA activities to total hours.
Purpose of the metric: firstly, to increase the planning accuracy, and secondly, to monitor and manage the team’s performance.
4. Accuracy of time estimation for areas \ types \ types of work
"Estimated working time" / "Actual working time"
Metric Assignment: allows you to use the correction factor for subsequent evaluations.
And finally, a group of metrics showing how the product was accepted by end users, how well it met their expectations. At the same time, as part of evaluating user interaction, not only feedback about the software itself is important to us. Another significant task of this group of metrics is to show whether users are satisfied with the process of interaction with the IT team in general and QA in particular.
1. User satisfaction with IT services
A regular survey of user satisfaction with IT services with scoring.
Purpose of the metric: to show whether users trust the IT team, whether they understand how and why their work is organized, how much this work meets expectations.
2. User satisfaction with the product
Regular user survey on how satisfied they are with the quality of the product.
Purpose of the metric: to determine how the product being developed meets user expectations, whether we are moving in that direction, whether we determine the importance of the features correctly and choose solutions.
3. Stakeholder Involvement
The number of initiatives and proposals for improving the process and product received during the iteration (release) by stakeholders.
Purpose of the metric: to determine the degree of participation of external stakeholders (business, infrastructure, users, support, etc.) in the work on the product. Having such a metric in your hands, you can navigate where you want to get feedback, so that one day you will not encounter problems and misunderstandings.
Pavel Novikov,
Program manager
https://doitsmartly.ru/
In this article I decided to gather together and consider the most basic, in my opinion, groups of criteria and measures for the QA process. And in each group I will list only the most important and illustrative, again, in my opinion, metrics as well as analyze what they are necessary for, in what situations are useful and how to use them.
What should be the metrics?
By itself, a metric in the context of software is a numerical expression of a property, the quality of the product itself or the process of its development. In other words, this is what we can use to measure, compare and evaluate software.
')
Now literally a couple of comments about the values ​​and properties of metrics:
- The main goal of any metric is to improve the development process and the software itself. The metric allows you to see at what point on the way to the goals we are at the moment, approach them or are removed, whether success criteria are achieved.
- Metrics should not exist for the sake of the measurement process itself. It is necessary to use only those metrics that really have practical value and will affect the further development of the product or process optimization. A simple rule follows from this: first you need to define zones for change / improvement, and then decide how to evaluate them.
Primary Metric Groups for QA
It is theoretically possible to invent your own characteristic, formula or indicator for practically every, even the most insignificant action, stage or status of the QA process. You can take into account each artifact, all transitions of defects by status, calculate the number of tests in the set. However, the most important question that you should immediately ask yourself is when there is a desire to measure something: “Why is this information needed, how can it be used?”. When forming a set of metrics, one should make a start from goals, plans for improving processes and product.
So, in this article we will not consider the usual quantitative indicators of the progress of testing, which are used in most reports and statuses. Instead, we will analyze which areas, artifacts and areas of development from the point of view of Quality Assurance, we must measure and control in order to assess the quality of work performed. Analysis and optimization of these points are extremely important for effective interaction with stakeholders ( https://doitsmartly.ru/all-articles/sw-testing/120-stakeholders-for-qa.html ) and software development in general:
1. Requirements for the software being developed.
Absolutely, we must understand that we are developing and testing, and the degree of this understanding must be able to be assessed. Potential risks or missed problems at the specification level can lead to the most serious and expensive errors.
2. The quality of the developed product.
Everything is obvious: you must be able to assess the quality of development and software in order to make predictions and assess risks. It is important to understand how high-quality and reliable the product is, relying not only on the presence or absence of errors found, but just predicting whether there are many potential problems.
3. Capabilities of the QA team.
It is also simple here: in order to manage the testing process, plan work and predict deadlines, it is always necessary to have not only the actual status of the tasks, but also the capabilities of the QA team.
4. The quality of the testing team.
In addition to the quality of the product itself, it is necessary to measure the effectiveness of the QA process itself and the testing team. To constantly optimize and improve the quality of work, you need to know where we are now, what allows us to move forward and what is thrown back.
5. Feedback and product satisfaction.
The latter is an area, but, of course, not in importance, but in the opinion of the stakeholders of the process, consumers of our services, users of the product. It is very important to be able to measure the overall degree of satisfaction with the product, highlight trends and draw appropriate conclusions. Properly selected metrics for this group will allow you to identify possible problems in time and promptly apply feedback to improve processes.
Next, we consider exactly which metrics are included in each group, analyze how exactly they can be evaluated. For each group, I will give a few examples of possible metrics and describe their meaning. These and some other QA process indicators are discussed in more detail in my article “The Most Important QA Metrics” ( https://doitsmartly.ru/all-articles/sw-testing/133-the-most-important-metrics-in-qa. html ).
Group 1 - Requirements for software being developed
This group of metrics will allow you to assess how much we have worked out the requirements (user story) for software, identify vulnerabilities and the most complex, potentially problematic software features, and understand where special control is required.
1. Test coverage requirements
"Total number of tests" / "Total number of requirements"
The purpose of the metric: to identify weaknesses in the test coverage, highlight the risks.
- This metric will work only if the requirements are well decomposed into equivalent ones. Otherwise, it will turn into an indicator of the presence or absence of tests for each of the requirements.
- For requirements where the coefficient is equal to or close to 0, the possibility of adding tests should be considered.
2. The degree of interconnectedness of requirements
AVERAGE (“Number of requirements related to requirement No. 1” / “Total number of requirements -1”, ..., “Number of requirements related to requirement No.n” / “Total number of requirements -1”)
The metric is calculated as the number of links of each requirement with the remaining requirements. It uses the average value for all requirements.
Purpose of the metric: to provide a basis for assessing the timing of testing and taking into account possible risks. Knowing the degree of mutual influence of requirements on each other, you can, for example, plan additional time and cases for end-to-end testing, work out regression tests, look towards integration, etc.
- From my own experience, I can say that the acceptable degree of connectedness does not exceed 0.2-0.3. Otherwise, the refinement of one of the requirements will lead to a chain of processing, and therefore possible errors in a significant part of the product.
3. The coefficient of stability requirements
“Number of changes in existing requirements” / “Total number of requirements implemented per iteration, including new ones”
Purpose of the metric: to show how many requirements already implemented have to be redone from release to release when developing new features. Also, the metric gives an idea of ​​how easily the system's functionality is scaled, new features are added.
- For this metric, the coefficient must be at least less than 0.5. In this case, we introduce new features in 2 times more than rework existing ones. Otherwise, the team focuses more on redoing previously released features, rather than creating new values ​​for the business.
Group 2 - Quality of the developed product
This group of metrics allows you to evaluate and compare from release to release both the quality of the software and the quality of the development itself.
1. The density of defects
“Number of defects in a separate module” / “Total number of defects in software”
It is calculated as the proportion of defects in their total number falling on a separate module within an iteration or release.
Purpose of the metric: highlight which part of the software is the most problematic. This information will help in assessing and planning work with this module, as well as in risk analysis.
- This metric will help draw our attention to the problem module \ system \ functionality, where the proportion of defects is above average.
2. Regression coefficient
"The number of defects in the old functionality" / "The total number of defects, including the new functionality"
The purpose of the metric is: to show what the team’s efforts are going to do - whether it is engaged in more development and debugging of new features or spends most of the time on fixes in already existing parts of the software.
- For example, if the regression coefficient is greater than 0.5, this means that more than half of the time we spend on restoring previously functioning software functions. This situation needs to be corrected.
3. The ratio of re-open defects
"The number of re-detected defects" / "The total number of errors, including previously corrected and new ones"
The purpose of the metric: to assess the quality of the development and correction of defects, as well as the complexity of the product or a separate module.
- The closer the coefficient value is to 0, the less the old errors recur.
- At the same time, if the value is greater than 0.2-0.3, this may indicate either the technical complexity of the module, or problems in the architecture, or poor-quality error correction.
Group 3 - Opportunities and effectiveness of the QA team
The main task of this group of metrics is to express in numbers what the testing team is capable of. These indicators can and should be calculated and compared on a regular basis, analyzing trends, observing how certain changes influence the work of a team.
1. Speed ​​of operation of the QA command
"The number of story points for N iterations)" / "N"
It is calculated as the ratio of realized story points \ requirements \ user stories for several iterations \ sprints to the number of selected iterations.
Purpose of the metric: to express numerically the possibilities, the speed of the team’s work for the further planning of the scope of work and analysis of development trends. The metric allows you to monitor the speed of QA, to monitor which internal or external influences on the team affect this speed.
2. The average lifetime of the defect
"The total time to correct defects found" / "Number of defects"
The total time during which the defects found during the iteration or release were open to the sum of the defects.
Purpose of the metric: show how much time on average it takes to work with one defect: its registration, correction and reproduction. This indicator will allow us to estimate the time required for testing, to highlight areas of software with which the greatest difficulties arise.
- The lifetime of a defect is the entire time from its registration to closure minus all possible work interruptions. By showing defects with the longest correction time, the metric will allow revealing particularly complex and problematic modules or the “weak link” in the development team.
Group 4 - Quality of the testing team
The task of this set of metrics is to assess how well testers perform their tasks, determine the level of competence and maturity of the QA team. With such a set of indicators, you can compare a team with it at different time intervals or with other, external testing groups.
1. The effectiveness of tests and test kits
"The number of errors found" / "The number of cases in the test set"
Purpose of the metric: to show how many errors on average our cases can detect. This metric reflects the quality of the test design and helps to monitor the trend of its change.
- The “killing rate” of tests allows you to monitor the effectiveness of each of the test kits, as it changes over time. This will make it possible to supplement them with “fresh” tests in time.
2. The ratio of errors missed on the product
"The number of errors detected after the release of the release" / "The total number of errors detected before and after the release"
Purpose of the metric: to demonstrate the quality of testing and the efficiency of error detection - what proportion of defects was filtered, and what percentage went to the productive level.
- Of course, the allowable percentage of missed errors will depend on many factors, however, if it is> 0.1, then there is clearly a problem, because in this case, every tenth defect has fallen on the productive side and has led to software failures among users.
3. Real time QA team work
"Time spent on target QA activity" / "Total working hours of the team"
The ratio of time spent by the team directly on target QA activities to total hours.
Purpose of the metric: firstly, to increase the planning accuracy, and secondly, to monitor and manage the team’s performance.
- Target activities may include: analysis, design, evaluation, testing, work meetings, and much more, non-targeted - simple due to blockers, communication problems, inaccessibility of resources, etc.
- Of course, this metric will never be equal to 1. Practice shows that for effective commands, its value can be 0.5-0.6.
4. Accuracy of time estimation for areas \ types \ types of work
"Estimated working time" / "Actual working time"
Metric Assignment: allows you to use the correction factor for subsequent evaluations.
- The degree of accuracy of the assessment can be determined for the whole team or individual testers, for the entire system or for individual software modules.
Group 5 - Feedback and User Satisfaction
And finally, a group of metrics showing how the product was accepted by end users, how well it met their expectations. At the same time, as part of evaluating user interaction, not only feedback about the software itself is important to us. Another significant task of this group of metrics is to show whether users are satisfied with the process of interaction with the IT team in general and QA in particular.
1. User satisfaction with IT services
A regular survey of user satisfaction with IT services with scoring.
Purpose of the metric: to show whether users trust the IT team, whether they understand how and why their work is organized, how much this work meets expectations.
- A metric can serve as an indicator of the need to focus on optimizing the process or to make it clearer and more transparent for users.
- The satisfaction rate can be calculated based on the results of the survey on the results of software delivery. For this you need to collect all the estimates and calculate the average score.
2. User satisfaction with the product
Regular user survey on how satisfied they are with the quality of the product.
Purpose of the metric: to determine how the product being developed meets user expectations, whether we are moving in that direction, whether we determine the importance of the features correctly and choose solutions.
- To calculate this metric, we also conduct a user survey and calculate the average score. By calculating this indicator on a regular basis, you can follow the trend of user satisfaction.
3. Stakeholder Involvement
The number of initiatives and proposals for improving the process and product received during the iteration (release) by stakeholders.
Purpose of the metric: to determine the degree of participation of external stakeholders (business, infrastructure, users, support, etc.) in the work on the product. Having such a metric in your hands, you can navigate where you want to get feedback, so that one day you will not encounter problems and misunderstandings.
Pavel Novikov,
Program manager
https://doitsmartly.ru/
Source: https://habr.com/ru/post/340192/
All Articles