Looking for the best way to test a system
What are the main approaches to testing, what are their strengths and weaknesses? Jan Jaap Kannegiter talks about how to determine which method is best used in each case.
The article is based on the speech of Jan at the June Heisenbug 2017 conference in St. Petersburg . Yang has been testing for over twenty years. He currently works as a test manager for acceptance tests at the Dutch standardization company Squerist. This report is about a variety of approaches to testing: from detailed elaboration of scenarios to research tours, from session testing to searching for errors together with users. His goal is to help you improve your professionalism, drawing attention to the methods that you have not used so far.
I would like to start with Linda Fisher. You do not know her, but more than 19 years ago I took her courses on testing. She said that testing needs to be done like this: first it, then that (interfaces, design, etc.).
')
Indeed, in those days, that was what was called testing. I put into practice what she taught me, and was successful in this. Today, after almost 20 years, I changed my mind about testing. I believe that there is no single approach that would work for all and give the opportunity to conduct the best testing. The methods chosen depend on your projects, on the situation, on your organization, on the system you work with.
I will talk about different approaches to testing and try to teach you to determine which one is best for your case.
Twenty years ago, the development of almost all systems was about the same. There were no major differences between the systems, as in our days. Here are the main changes that have occurred since:
There are two main types of testing - scripted and exploratory.
The main feature of scenario testing is that you start by dividing the task into stages (preparation, execution, completion, etc.) and then try to perform all the actions according to these stages. That is, this approach can be described as "I think on this in advance and then execute."
Research testing involves a completely different approach - the development of the testing process occurs directly during operation. Testing starts from an important point, other important points of testing appear in the process, and each time the specialist decides which of them is most important and in which direction to go next. That is, thinking through and running tests take place in constant interaction.
Some people rely entirely on scenario testing and find exploratory testing dangerous. Others, on the contrary, use exploratory testing and consider the scenario to be something of the past. I think both are right and wrong. Both approaches may be valuable, but this depends on the situation.
However, these two approaches to testing are not the only ones. In addition to them, there are other approaches that are somewhere in between.
Why did I arrange them in that order? Research testing requires virtually no training, and for scenario testing, you need to seriously prepare. And, for example, the session is somewhere in the middle - it requires preparation, but not so big.
A professional tester should be aware of each of these approaches and understand which one is best for each specific case.
Let us examine these directions in testing in more detail.
What are the differences between detailed scenario and general scenario testing? I will explain with an example.
If we are testing, say, mail, then detailed script testing may look like this:
For the same example, a general test script might look like this:
If the person who prepares the tests and performs them is the same person, it makes no sense to make them too detailed. Therefore, consider whether it is worth spending so much time preparing a detailed script test? Perhaps a general testing script would be enough.
In those days, when research testing only appeared, many did not understand what testers are doing. They began work without planning, without explaining what they were going to do and how. Therefore, for many people, research testing seemed like one huge cloud. Later, someone smart decided to divide this cloud into small clouds corresponding to the sessions. So there was a session test.
When using this approach, the tester has testing points with which he is going to work during one session, a small amount of documentation that he has to follow. But at the same time, he has much more freedom than in the case of detailed script testing, and this makes him happy (that is why I put smiles on the illustrations).
Session testing provides for:
Searching for bugs has a lot in common with session testing, but there are important differences. Not only testers but also developers as well as users take part in the search for bugs.
A bug search session can last longer than a testing session. So, for a session of searching for bugs, the duration is usually from three to four hours, and during session testing after two hours, as a rule, it is necessary to take a break. In addition, during the search for bugs, work is usually done in pairs (tester plus user), and the purpose of such a session is not only obtaining information, but also approving the system by users.
On the Internet, you can read that research testing is such a testing technique. In fact, this is not just a technique, but a full-fledged approach to testing, which allows you to perform complete system testing.
Some say research testing has no structure. This is also not true. After testing is complete, you can have as much documentation as you want, you just do not develop it in advance.
When conducting research testing focuses on personal freedom and responsibility of the tester. It implies a constant search for answers to the questions: “how to do better?”, “Which tests are the most important now?”. You are not just doing what is written in the script, you are constantly thinking. In addition, all the stages of testing (design, implementation, interpretation of results, etc.) are not carried out sequentially, but in parallel throughout the project.
When conducting research testing, it is sometimes difficult to focus on something specific. And in this case, help research tours. Tours reflect the main goals and objectives that are set during testing.
In this regard, I would like to quote from James Whittaker: “ Research testing without good guidance is like wandering around the city in search of interesting sights. The manual provides an opportunity to understand where you need to follow . ”
In the book, "Testing Software Testing," James Whittaker writes about a wide variety of research tours. Here are some of them:
So, the main differences between research testing and scenario testing are presented in the table below:
Every time I do this presentation, I make the same mistake - everyone has the feeling that exploratory testing is better than script testing. This is not true. Scenario testing is just as valuable as research, but it all depends on the situation.
However, the effectiveness of research testing is still higher than the scenario. This is proved by various studies. In particular, in 2007 a study was conducted in which two teams participated. They simultaneously worked on testing the same application, the first using scenario testing, and the second research. Then they tested the second application, but now the first team used research testing, and the second one - scenario testing. This study showed that, in exploratory testing, error detection performance was much higher. Similar results were obtained in other studies of the effectiveness of test methods.
From my own experience, I can say that many factors influence the choice of testing approach. Among them: time, budget, goals, testing objectives, testers skills, system, development method, documentation, type of organization.
In the table below, I present some features of organizations and systems, as well as common goals. You can see which method is best suited for each situation, but it is worth bearing in mind that in many cases it makes sense to apply both methods, even if this is not indicated in the table. Always try to think critically and adapt the information to your situation.
So which testing approach works best? The answer is obvious: it all depends on the situation. But I believe that in future software testing will be a combination of automated and research testing. Therefore, each specialist should study, know well and apply both methods.
If your professional activity is related to testing, you will surely be interested in these reports at our two-day December conference Heisenbug 2017 Moscow :
The article is based on the speech of Jan at the June Heisenbug 2017 conference in St. Petersburg . Yang has been testing for over twenty years. He currently works as a test manager for acceptance tests at the Dutch standardization company Squerist. This report is about a variety of approaches to testing: from detailed elaboration of scenarios to research tours, from session testing to searching for errors together with users. His goal is to help you improve your professionalism, drawing attention to the methods that you have not used so far.
Why are there different approaches to testing?
I would like to start with Linda Fisher. You do not know her, but more than 19 years ago I took her courses on testing. She said that testing needs to be done like this: first it, then that (interfaces, design, etc.).
')
Indeed, in those days, that was what was called testing. I put into practice what she taught me, and was successful in this. Today, after almost 20 years, I changed my mind about testing. I believe that there is no single approach that would work for all and give the opportunity to conduct the best testing. The methods chosen depend on your projects, on the situation, on your organization, on the system you work with.
I will talk about different approaches to testing and try to teach you to determine which one is best for your case.
Twenty years ago, the development of almost all systems was about the same. There were no major differences between the systems, as in our days. Here are the main changes that have occurred since:
- Flexibility in development prevails over clear planning.
- We focus more on skill than on methods.
- Goals are more important than process. If earlier we followed the two hundred page manual, which described the entire testing process step by step, today it is not necessary. It is important to see what result we want to achieve, and do everything for this.
- Rapid testing prevails over solid. Most organizations want to enter the market as quickly as possible.
- Pragmatic methods prevail over standards. Following the standards can increase the testing time, the amount of required documentation and the required staff more than three times. Therefore, even in organizations that are engaged in standardization, they often refuse to test standards.
The main approaches to testing
There are two main types of testing - scripted and exploratory.
The main feature of scenario testing is that you start by dividing the task into stages (preparation, execution, completion, etc.) and then try to perform all the actions according to these stages. That is, this approach can be described as "I think on this in advance and then execute."
Research testing involves a completely different approach - the development of the testing process occurs directly during operation. Testing starts from an important point, other important points of testing appear in the process, and each time the specialist decides which of them is most important and in which direction to go next. That is, thinking through and running tests take place in constant interaction.
Some people rely entirely on scenario testing and find exploratory testing dangerous. Others, on the contrary, use exploratory testing and consider the scenario to be something of the past. I think both are right and wrong. Both approaches may be valuable, but this depends on the situation.
Intermediate approaches to testing
However, these two approaches to testing are not the only ones. In addition to them, there are other approaches that are somewhere in between.
Why did I arrange them in that order? Research testing requires virtually no training, and for scenario testing, you need to seriously prepare. And, for example, the session is somewhere in the middle - it requires preparation, but not so big.
A professional tester should be aware of each of these approaches and understand which one is best for each specific case.
Let us examine these directions in testing in more detail.
Detailed scripting and general scripting (Detailed Scripting & Global scripting)
What are the differences between detailed scenario and general scenario testing? I will explain with an example.
If we are testing, say, mail, then detailed script testing may look like this:
- Go to the start page
- Click on the "New" button
- In the "To" field, write j.cannegieter@squerist.nl
- Click on the "File"
- Select the document "Test1"
- In the "Subject" field write "Test attachment <date>"
- Open jcannegieter mailbox
- Check if the message and the attachment is delivered.
For the same example, a general test script might look like this:
- Create and send an email with an attachment to one recipient
- Check if message and attachment is delivered
If the person who prepares the tests and performs them is the same person, it makes no sense to make them too detailed. Therefore, consider whether it is worth spending so much time preparing a detailed script test? Perhaps a general testing script would be enough.
Session Testing and Search for Bugs (Session Based Testing & Bug Hunts)
In those days, when research testing only appeared, many did not understand what testers are doing. They began work without planning, without explaining what they were going to do and how. Therefore, for many people, research testing seemed like one huge cloud. Later, someone smart decided to divide this cloud into small clouds corresponding to the sessions. So there was a session test.
When using this approach, the tester has testing points with which he is going to work during one session, a small amount of documentation that he has to follow. But at the same time, he has much more freedom than in the case of detailed script testing, and this makes him happy (that is why I put smiles on the illustrations).
Session testing provides for:
- Sessions testing.
- Missions and points of testing / ideas for testing.
- Notes during the session.
- Report after the session (what we found, what is the quality of the system, etc.).
Searching for bugs has a lot in common with session testing, but there are important differences. Not only testers but also developers as well as users take part in the search for bugs.
A bug search session can last longer than a testing session. So, for a session of searching for bugs, the duration is usually from three to four hours, and during session testing after two hours, as a rule, it is necessary to take a break. In addition, during the search for bugs, work is usually done in pairs (tester plus user), and the purpose of such a session is not only obtaining information, but also approving the system by users.
Exploratory Testing & Test Tours
On the Internet, you can read that research testing is such a testing technique. In fact, this is not just a technique, but a full-fledged approach to testing, which allows you to perform complete system testing.
Some say research testing has no structure. This is also not true. After testing is complete, you can have as much documentation as you want, you just do not develop it in advance.
When conducting research testing focuses on personal freedom and responsibility of the tester. It implies a constant search for answers to the questions: “how to do better?”, “Which tests are the most important now?”. You are not just doing what is written in the script, you are constantly thinking. In addition, all the stages of testing (design, implementation, interpretation of results, etc.) are not carried out sequentially, but in parallel throughout the project.
When conducting research testing, it is sometimes difficult to focus on something specific. And in this case, help research tours. Tours reflect the main goals and objectives that are set during testing.
In this regard, I would like to quote from James Whittaker: “ Research testing without good guidance is like wandering around the city in search of interesting sights. The manual provides an opportunity to understand where you need to follow . ”
In the book, "Testing Software Testing," James Whittaker writes about a wide variety of research tours. Here are some of them:
- Feature Tour - Test all application functionality.
- Requirement Tour - test to see if commercial requirements are met.
- Data tour - test data processing.
- Computation Tour - Test all calculations.
- Process Tour - test application support for workflows.
- Screen Tour - test all screens.
- Tour of the bad parts - test those parts of the application that previously had a lot of errors.
- Interface Tour - Test all interfaces.
- Asocial tour - test incorrect use.
- Tour robber - to test on the subject of whether you can get unauthorized access.
The main differences of testing methods and comparison of their effectiveness
So, the main differences between research testing and scenario testing are presented in the table below:
| Scenario testing | Research testing |
| Focused on preparation | Focused on action |
| Focused on planning | Flexible |
| Based on methods | Pragmatic |
| Obeys the process | Puts the tester in the spotlight |
| Focused on documentation | Focused on performing tests |
Every time I do this presentation, I make the same mistake - everyone has the feeling that exploratory testing is better than script testing. This is not true. Scenario testing is just as valuable as research, but it all depends on the situation.
However, the effectiveness of research testing is still higher than the scenario. This is proved by various studies. In particular, in 2007 a study was conducted in which two teams participated. They simultaneously worked on testing the same application, the first using scenario testing, and the second research. Then they tested the second application, but now the first team used research testing, and the second one - scenario testing. This study showed that, in exploratory testing, error detection performance was much higher. Similar results were obtained in other studies of the effectiveness of test methods.
How to decide which test method is best suited
From my own experience, I can say that many factors influence the choice of testing approach. Among them: time, budget, goals, testing objectives, testers skills, system, development method, documentation, type of organization.
In the table below, I present some features of organizations and systems, as well as common goals. You can see which method is best suited for each situation, but it is worth bearing in mind that in many cases it makes sense to apply both methods, even if this is not indicated in the table. Always try to think critically and adapt the information to your situation.
| Category | Situation | Scenario | Research |
| System | Many calculations | + | |
| Interface oriented | + | ||
| Backend oriented | + | ||
| Mobile app | + | ||
| Test objectives | Compliance Check | + | |
| System Validation | + | ||
| Usability | + | ||
| Testing business rules | + | ||
| Performance | + | ||
| Automation check | + | ||
| Security | + | + | |
| Organization | Focuses on planning and preparation | + | |
| Energetic modern startup | + | ||
| Hierarchical, traditional | + | ||
| Welcomes the municipality | + | + | |
| Documentation | Many detailed documentation | + | + |
| Some documentation | + | ||
| Requirements / documentation are constantly changing. | + | ||
| Development | Cascade model | + | + |
| Agile | + | ||
| Budget | Big | + | + |
| Small | |||
| Time | Inclusion in the work in the early stages | + | + |
| Inclusion in work on late terms | + | ||
| A lot of time | + | + | |
| Little time | + | ||
| Testers Skills | Analytical, scrupulous | + | |
| Critical thinking (doubt everything) | + | ||
| Flexibility | + | ||
| Professional testers | + | + | |
| Non-professional testers | + | + |
Instead of conclusion
So which testing approach works best? The answer is obvious: it all depends on the situation. But I believe that in future software testing will be a combination of automated and research testing. Therefore, each specialist should study, know well and apply both methods.
If your professional activity is related to testing, you will surely be interested in these reports at our two-day December conference Heisenbug 2017 Moscow :
- Automation in Gamedev: how WorldOfTanks bots were raised (Alexander Shukov, Wargaming.net)
- Pandora's White Box (Nikita Makarov, Odnoklassniki)
- TestContainers: Java applications integration testing (Anton Arkhipov, ZeroTurnaround)
Source: https://habr.com/ru/post/341852/
All Articles