The method of calculating the coefficient reflecting the quality of the released software product
One of the main criteria for evaluating a released software product is its quality. The quality actually shows how well programmers and testers coped with their task and how much the released product is ready for real use.
Unfortunately, for a number of reasons, released products always contain defects that were not found at the testing / development stage. Most of them manifest themselves as a result of previously unaccounted use cases, non-envisaged use cases, conflicts with other software products in the work environment. In addition, errors can be the result of poor performance of the Quality Assurance department or the negligence of developers.
In this regard, the question arises how to assess the quality of the product and understand how well the product development team has done its job.
We set ourselves the task of counting the numerical expression of the overall quality of the delivered system and, as a result, we developed a system of counting numbers (Quality Score) that would uniquely determine how well the team coped with its work.
Traditionally, errors in accounting systems (bug track system) are divided into categories:
Critical
An error that does not allow using some system functionality and cannot be leveled by changing settings or other user actions.
Major
An error that makes it difficult to use the system but which can be leveled by changing the settings or using an alternative scenario of work
Minor
An error that makes it difficult to use the system but does not actually affect the way of working
Trivial
An error that does not actually affect the use of the system function but affects the overall quality. For example, spelling errors or typographical errors.
')
At first glance, it seems possible to count the QS just to count the number of all errors found. But using this method we risk getting frightening numbers for a generally good enough product. This can occur due to the fact that most of the errors (even having the status of Critical and Major) will not interfere with the daily use of the product by most users.
As an example, you can look at the public Skype bug tracker (jira.skype.com). At the time of writing there was the following picture:
Critical - 242
Major - 151
That is, in general, with frightening figures, we have a completely workable product that millions of people use without special problems.
On the other hand, a situation is possible when the presence of even one error with the Critical status (for example, a failure in the installation process) will make the use of the system completely impossible.
As a result of the above considerations, we decided to create an additional classification of errors in accordance with the scenario in which it is used:
Standard usage scenario :
The usage scenario that was provided during the creation of the system.
In turn, all standard scripts are also classified as:
- Basic scripts
- Secondary scripts
- Rare scenarios
Non-standard usage scenario:
A scenario that was not provided during the development / testing phase.
In turn, are divided into:
- Option normal use
- Option not normal use
A scenario that was not provided for and which cannot be considered a normal way to use the system.
Each type of use case and each category of error is assigned a specific coefficient.
Basic scenario
Standard 12
Not standard 4
Background script
Standard 11
Not standard 3
Rare script
Standard 10
Not standard 2
Scenario not normal
use 1
Critical 20
Major 10
Minor 5
Trivial 2
The coefficient for a specific defect is calculated as the coefficient of the category of the scenario in which the error occurred for the coefficient of severity of the error.
The resulting coefficient is calculated as the sum of the coefficients for all errors.
As a result, we have QS which determines the quality of the system. Smaller values - greater quality. The ideal quality is 0.
Example:
- System with 2 Crtitical defects in standard baseline use cases and 2
Major defects in abnormal use cases will have
QS = 500 (2 * 20 * 12 + 2 * 10 * 1)
- A system with 1 Major defective in a non-standard secondary use case and 2 Trivial defects in standard rare use cases will have QS = 70 (1 * 10 * 3 + 2 * 2 * 10)
Unfortunately, for a number of reasons, released products always contain defects that were not found at the testing / development stage. Most of them manifest themselves as a result of previously unaccounted use cases, non-envisaged use cases, conflicts with other software products in the work environment. In addition, errors can be the result of poor performance of the Quality Assurance department or the negligence of developers.
In this regard, the question arises how to assess the quality of the product and understand how well the product development team has done its job.
We set ourselves the task of counting the numerical expression of the overall quality of the delivered system and, as a result, we developed a system of counting numbers (Quality Score) that would uniquely determine how well the team coped with its work.
Traditionally, errors in accounting systems (bug track system) are divided into categories:
Critical
An error that does not allow using some system functionality and cannot be leveled by changing settings or other user actions.
Major
An error that makes it difficult to use the system but which can be leveled by changing the settings or using an alternative scenario of work
Minor
An error that makes it difficult to use the system but does not actually affect the way of working
Trivial
An error that does not actually affect the use of the system function but affects the overall quality. For example, spelling errors or typographical errors.
')
At first glance, it seems possible to count the QS just to count the number of all errors found. But using this method we risk getting frightening numbers for a generally good enough product. This can occur due to the fact that most of the errors (even having the status of Critical and Major) will not interfere with the daily use of the product by most users.
As an example, you can look at the public Skype bug tracker (jira.skype.com). At the time of writing there was the following picture:
Critical - 242
Major - 151
That is, in general, with frightening figures, we have a completely workable product that millions of people use without special problems.
On the other hand, a situation is possible when the presence of even one error with the Critical status (for example, a failure in the installation process) will make the use of the system completely impossible.
As a result of the above considerations, we decided to create an additional classification of errors in accordance with the scenario in which it is used:
Standard usage scenario :
The usage scenario that was provided during the creation of the system.
In turn, all standard scripts are also classified as:
- Basic scripts
- Secondary scripts
- Rare scenarios
Non-standard usage scenario:
A scenario that was not provided during the development / testing phase.
In turn, are divided into:
- Option normal use
- Option not normal use
A scenario that was not provided for and which cannot be considered a normal way to use the system.
Each type of use case and each category of error is assigned a specific coefficient.
Basic scenario
Standard 12
Not standard 4
Background script
Standard 11
Not standard 3
Rare script
Standard 10
Not standard 2
Scenario not normal
use 1
Critical 20
Major 10
Minor 5
Trivial 2
The coefficient for a specific defect is calculated as the coefficient of the category of the scenario in which the error occurred for the coefficient of severity of the error.
The resulting coefficient is calculated as the sum of the coefficients for all errors.
As a result, we have QS which determines the quality of the system. Smaller values - greater quality. The ideal quality is 0.
Example:
- System with 2 Crtitical defects in standard baseline use cases and 2
Major defects in abnormal use cases will have
QS = 500 (2 * 20 * 12 + 2 * 10 * 1)
- A system with 1 Major defective in a non-standard secondary use case and 2 Trivial defects in standard rare use cases will have QS = 70 (1 * 10 * 3 + 2 * 2 * 10)
Source: https://habr.com/ru/post/123561/
All Articles