What tests do you need? Part 1. On the goals and objectives that affect the choice of tests
This article is an introduction to the matrix of tests, which I actually want to share and share in the second part.
The article is focused on project managers who want to understand what kind of testing they need, and on testers who don't want to do a lot of work is not clear for what reason.
I hope that this will also be useful for newcomers to QA and those who want to move up the career path to test managers or lead tests.
')
It is my deep conviction that the main reason for the low efficiency of the software development teams is the desire to solve a problem without seeing the final goal . If you constantly keep a goal in your head as an answer to the question “why do we do this?”, You can not only solve the problem so that you do n’t have to be redone three hundred times, but you don’t waste time on tasks that do not lead to goals at all.
Here, many will say that setting tasks is the business of managers, and that they should have goals in mind, and we, like puppets, should press buttons when the TODO thread is twitching. In the Perfect World, this is true. But in our world, leaders are just people, as imperfect as all of us.
Just insure each other against mistakes, you are a team!
And another important thing. The task set by the head. And he will ask you the result, measured by him with the help of some kind of performance indicators. That is, staging from it, and efficiency - from you. I have already stressed that understanding goals affects performance. So this is really your business.
Why am I talking about this? Moreover, it often happens that performing the task “conduct full manual functional testing” does not lead to an increase in the efficiency of the development team. It takes a lot of resources, and the result is low, and the task is almost never fully completed. How often do you think that you really need something else? Let me guess how it ended! Hire more monkey-clickers or automate regression. On this your thoughts, for sure, often stopped.
I propose to think about what tests you really need based on the variety of possible goals.
By project team, I mean developers, testers and managers who are together engaged in the process of creating a software product.
By goal, I mean a common goal for the whole team.
And the tasks in this article I consider only the testing team.
Imagine a task tree. Upstairs Goal. Next come the tasks that I call the “higher level”. It is these tasks that “come from above”, go from the head. Each top-level task is decomposed to an arbitrary depth. In this process can be involved as performers, and managers, depending on the independence of subordinates and the hierarchy of the team.
High-level tasks are always strategic , as they directly affect the achievement of the goal. The rest are tactical.
Defining a goal is a difficult process. There are many methods and techniques that project managers should learn. Here I will only briefly mention the variety of goals that can emerge at the exit from the competent process of defining the goal. This is necessary in order for you to understand that each project team has a different goal from the other team’s goal, and it’s likely that you need to set tasks to achieve a goal.
The goal of the project team depends on several factors at the same time. Suppose from the following:
Take the blurry wording "improve the quality of software." The goal looks the same, but for different projects it will mean different things.
The project exists recently, the team works from scratch. There is no sale at all, the prototype is released on a demo stand, which end users do not have access to. Damage from instability - loss of investor interest. All defects found, except for minor or "improvements", fix immediately. The customers themselves set the requirements themselves, but investors dictate the weather. "Increase quality" for such a command would mean the following:
It can be said that this list is the top-level strategic tasks that must “come from above” or be put together by the team.
Let's look at the point “to make so that on the demo stand a new release does not fall on the basic demo scenario”. It seems obvious that this is the same as setting the task “conduct a Smoke test”. But this is not true. Testers will determine for themselves what their Smokey test will be and how to achieve stability in the baseline scenario. Perhaps it will be generally different processes. Or maybe there will be some smoke tests. For example, a checklist that checks the correctness of settings in a modified stand configuration + some queries to the database can be sufficient checks that the script will be successful, since the UI has not changed in the last iteration. Or maybe this scenario will be included in the regression test. Performers will decide all this, seeing the full set of goals and strategic objectives.
Project a few years. The team, which was engaged in customization and implementation, transferred the code to support another team. There is a kernel update schedule supplied to all customers and received from the vendor. Hotfixes should be uploaded for 2 days. There are penalties for violation of the terms. There are open defects that have not yet been discovered by the customer, but are acquired within the team. Some of the defects are already in sale, some have appeared after receiving a new delivery from the vendor. If the customer is not satisfied, he will not extend the license for the next year, but will go to competitors.
Take the same goal: "improve the quality of software." Strategic objectives:
Do you see the phrase “regress” in this list, or “make 100% coverage” or “automate the smoke”? Not. Do not need. Such tasks should be set when decomposing tasks and not losing sight of the Objective. If you omit strategic tasks that seem “obvious” and “self-evident” to you, and leave only the naked Goal “to improve quality” - and a number of requirements about exactly what tools to do it - you will miss the opportunity to achieve the goal effectively, or not you will never achieve anything yet, and the customer will go to a competitor, or a startup will be closed due to lack of funding. There is also a middle ground - to remain in the current position, periodically scrubbing from the higher leadership and constantly experiencing a collective sense of guilt and own incompetence.
If there are no strategic goals, and you are a subordinate, and not a leader, formulate them yourself. Such tasks will help find an effective solution and understand what tests you need .
There are factors that do not affect the goal setting, and the problem statement for testers is still how.
The most obvious factor. Under the form implies architecture, components. Is there a UI? Is there a database? Web application? Many services that run independently?
Depending on the first factor, the entire set of possible tasks for testing is reduced to inapplicable (for example, we do not test the usability of services launched by the task scheduler) and many mandatory ones are allocated (we will check for vulnerabilities for the web interface of a public application). But all this concerns the objects of testing . as part of the “conduct functional testing” task, a number of necessary and sufficient tests are determined by the performers when creating a test plan. Knowledge of how to define this set is within the competence of QA-specialists. This means that these factors do not affect not only the definition of the goal, but also the formulation of top-level tasks.
Looking at these characteristics is determined by the need to conduct testing of fault tolerance, performance and scaling. These are top-level tasks right below the goal.
In addition, the strategy for the preparation of test data and scenarios depends on the amount of data. On the one hand, this is the competence of the performers, and on the other hand, it is necessary to set aside for this time, and generally set the task to think it over. You may need to use data from the pro-environment and their depersonalization, and this should be ensured by the manager.
Part of the testing tasks can not be performed without the participation of developers. For example, unit tests generally should ideally be written by them. For integration testing, for example, knowledge of the logical interfaces between the components is necessary, and only developers can provide this information. It is often difficult to write queries to the database for checking test data without knowing the data model and without having any documentation on this. Tracking the test code coverage is possible only by joint efforts. And so on.
If the relationship with the developers is not established, some types of testing will not be available in principle. And all the tests will be black box tests and nothing more.
Sometimes the lack of interaction with developers is compensated by the completeness of the documentation. And sometimes the opposite - the developers will tell what is not written.
The availability and completeness of the documentation determines the readiness of the tests - ad hock or on a test model.
Well, the documentation itself can be the object of testing. No specs - no specification testing.
Little point in autotests if there is no CIS. There will be no white box testing if there is no access to the sources via VCS. Load tests generally depend entirely on what to drive them.
Bare hands are actually very little that can be done.
There was such a case. For testing, a simulator was used that could not reproduce some situations. It was necessary to simulate the behavior of an external system by changing the input data in the application under test. To do this, I had to go into the code and enter the transformations there. Very dirty and dishonest method, which gives the error of the method, but otherwise it was not how. This example refers to the toolkit. But the problem was solved not by the presence of the tool, but by the possibility of getting into the code.
If you can get into the code - you can localize the detected defect. You can reduce the number of defects that appeared as a result of incorrect configuration of the test bench, and not due to an error in the logic of the system. You can run a code review of a small piece of code and return it immediately, without wasting time playing complex scenarios to test it. Such tests will be white box tests.
Surely, there are still some factors that I was too lazy to remember now. Write in the comments, please!
The article is focused on project managers who want to understand what kind of testing they need, and on testers who don't want to do a lot of work is not clear for what reason.
I hope that this will also be useful for newcomers to QA and those who want to move up the career path to test managers or lead tests.
Lyrical digression
')
It is my deep conviction that the main reason for the low efficiency of the software development teams is the desire to solve a problem without seeing the final goal . If you constantly keep a goal in your head as an answer to the question “why do we do this?”, You can not only solve the problem so that you do n’t have to be redone three hundred times, but you don’t waste time on tasks that do not lead to goals at all.
Here, many will say that setting tasks is the business of managers, and that they should have goals in mind, and we, like puppets, should press buttons when the TODO thread is twitching. In the Perfect World, this is true. But in our world, leaders are just people, as imperfect as all of us.
Therefore, I urge both leaders and performers to think about the Goals.
Just insure each other against mistakes, you are a team!
And another important thing. The task set by the head. And he will ask you the result, measured by him with the help of some kind of performance indicators. That is, staging from it, and efficiency - from you. I have already stressed that understanding goals affects performance. So this is really your business.
Why am I talking about this? Moreover, it often happens that performing the task “conduct full manual functional testing” does not lead to an increase in the efficiency of the development team. It takes a lot of resources, and the result is low, and the task is almost never fully completed. How often do you think that you really need something else? Let me guess how it ended! Hire more monkey-clickers or automate regression. On this your thoughts, for sure, often stopped.
I propose to think about what tests you really need based on the variety of possible goals.
Conditional terminology.
By project team, I mean developers, testers and managers who are together engaged in the process of creating a software product.
By goal, I mean a common goal for the whole team.
And the tasks in this article I consider only the testing team.
Imagine a task tree. Upstairs Goal. Next come the tasks that I call the “higher level”. It is these tasks that “come from above”, go from the head. Each top-level task is decomposed to an arbitrary depth. In this process can be involved as performers, and managers, depending on the independence of subordinates and the hierarchy of the team.
High-level tasks are always strategic , as they directly affect the achievement of the goal. The rest are tactical.
Many goals and objectives
Defining a goal is a difficult process. There are many methods and techniques that project managers should learn. Here I will only briefly mention the variety of goals that can emerge at the exit from the competent process of defining the goal. This is necessary in order for you to understand that each project team has a different goal from the other team’s goal, and it’s likely that you need to set tasks to achieve a goal.
The goal of the project team depends on several factors at the same time. Suppose from the following:
- How long is the project?
- The frequency of deliveries releases in prod.
- Damage from system instability.
- Number and criticality of known unclosed defects.
- Relationship with the customer.
Take the blurry wording "improve the quality of software." The goal looks the same, but for different projects it will mean different things.
Example 1. Startup.
The project exists recently, the team works from scratch. There is no sale at all, the prototype is released on a demo stand, which end users do not have access to. Damage from instability - loss of investor interest. All defects found, except for minor or "improvements", fix immediately. The customers themselves set the requirements themselves, but investors dictate the weather. "Increase quality" for such a command would mean the following:
- make sure the build doesn't crash
- make it so that on the demo stand a new release does not fall on the basic demo scenario
- continue to have time to fix all the important defects in the new functionality before the release
- monitor the code coverage with tests, increasing the coverage as the functional increments
- Each new release fixes several tasks from the category of "improvements", gradually reducing the number of unclosed tickets of this class to zero.
- with all this, continue to have time to spend most of the time on the increment of the functional, which is required to be fully checked before the release.
It can be said that this list is the top-level strategic tasks that must “come from above” or be put together by the team.
Let's look at the point “to make so that on the demo stand a new release does not fall on the basic demo scenario”. It seems obvious that this is the same as setting the task “conduct a Smoke test”. But this is not true. Testers will determine for themselves what their Smokey test will be and how to achieve stability in the baseline scenario. Perhaps it will be generally different processes. Or maybe there will be some smoke tests. For example, a checklist that checks the correctness of settings in a modified stand configuration + some queries to the database can be sufficient checks that the script will be successful, since the UI has not changed in the last iteration. Or maybe this scenario will be included in the regression test. Performers will decide all this, seeing the full set of goals and strategic objectives.
Do not set the frame, saying "how to do." Say "what you need to achieve."
Example 2. Support of the ERMS integrated into the infrastructure of several enterprises.
Project a few years. The team, which was engaged in customization and implementation, transferred the code to support another team. There is a kernel update schedule supplied to all customers and received from the vendor. Hotfixes should be uploaded for 2 days. There are penalties for violation of the terms. There are open defects that have not yet been discovered by the customer, but are acquired within the team. Some of the defects are already in sale, some have appeared after receiving a new delivery from the vendor. If the customer is not satisfied, he will not extend the license for the next year, but will go to competitors.
Take the same goal: "improve the quality of software." Strategic objectives:
- Find and fix as many defects as possible before customers notice them.
- Minimize the number of known defects that are in the sale.
- Strive to ensure that the new supply does not contain defects at all.
- Always have time to correct defects found by customers in a timely manner.
Do you see the phrase “regress” in this list, or “make 100% coverage” or “automate the smoke”? Not. Do not need. Such tasks should be set when decomposing tasks and not losing sight of the Objective. If you omit strategic tasks that seem “obvious” and “self-evident” to you, and leave only the naked Goal “to improve quality” - and a number of requirements about exactly what tools to do it - you will miss the opportunity to achieve the goal effectively, or not you will never achieve anything yet, and the customer will go to a competitor, or a startup will be closed due to lack of funding. There is also a middle ground - to remain in the current position, periodically scrubbing from the higher leadership and constantly experiencing a collective sense of guilt and own incompetence.
Strategic objectives specify the goal, making it truly obvious.
If there are no strategic goals, and you are a subordinate, and not a leader, formulate them yourself. Such tasks will help find an effective solution and understand what tests you need .
What determines the choice of tests
There are factors that do not affect the goal setting, and the problem statement for testers is still how.
Factor 1. Type of developed application / information system
The most obvious factor. Under the form implies architecture, components. Is there a UI? Is there a database? Web application? Many services that run independently?
Depending on the first factor, the entire set of possible tasks for testing is reduced to inapplicable (for example, we do not test the usability of services launched by the task scheduler) and many mandatory ones are allocated (we will check for vulnerabilities for the web interface of a public application). But all this concerns the objects of testing . as part of the “conduct functional testing” task, a number of necessary and sufficient tests are determined by the performers when creating a test plan. Knowledge of how to define this set is within the competence of QA-specialists. This means that these factors do not affect not only the definition of the goal, but also the formulation of top-level tasks.
Factor 2. The amount of data and the number of users.
Looking at these characteristics is determined by the need to conduct testing of fault tolerance, performance and scaling. These are top-level tasks right below the goal.
In addition, the strategy for the preparation of test data and scenarios depends on the amount of data. On the one hand, this is the competence of the performers, and on the other hand, it is necessary to set aside for this time, and generally set the task to think it over. You may need to use data from the pro-environment and their depersonalization, and this should be ensured by the manager.
Factor 3. Developer Relations.
Part of the testing tasks can not be performed without the participation of developers. For example, unit tests generally should ideally be written by them. For integration testing, for example, knowledge of the logical interfaces between the components is necessary, and only developers can provide this information. It is often difficult to write queries to the database for checking test data without knowing the data model and without having any documentation on this. Tracking the test code coverage is possible only by joint efforts. And so on.
If the relationship with the developers is not established, some types of testing will not be available in principle. And all the tests will be black box tests and nothing more.
Factor 4. The presence and completeness of the specification.
Sometimes the lack of interaction with developers is compensated by the completeness of the documentation. And sometimes the opposite - the developers will tell what is not written.
The availability and completeness of the documentation determines the readiness of the tests - ad hock or on a test model.
Well, the documentation itself can be the object of testing. No specs - no specification testing.
Factor 5. Toolkit.
Little point in autotests if there is no CIS. There will be no white box testing if there is no access to the sources via VCS. Load tests generally depend entirely on what to drive them.
Bare hands are actually very little that can be done.
Factor 6. The ability to "climb into the code."
There was such a case. For testing, a simulator was used that could not reproduce some situations. It was necessary to simulate the behavior of an external system by changing the input data in the application under test. To do this, I had to go into the code and enter the transformations there. Very dirty and dishonest method, which gives the error of the method, but otherwise it was not how. This example refers to the toolkit. But the problem was solved not by the presence of the tool, but by the possibility of getting into the code.
If you can get into the code - you can localize the detected defect. You can reduce the number of defects that appeared as a result of incorrect configuration of the test bench, and not due to an error in the logic of the system. You can run a code review of a small piece of code and return it immediately, without wasting time playing complex scenarios to test it. Such tests will be white box tests.
Surely, there are still some factors that I was too lazy to remember now. Write in the comments, please!
Source: https://habr.com/ru/post/246211/
All Articles