Testing: Manual or Automated?
I want to share my experience in organizing the testing process, which covers 3 years of my work and the creation of several large systems. The description will only affect the automation of "manual" testing without interfering with other aspects of software development.
I think it is worth mentioning at once that at all stages we used:
Everywhere, where I will talk about testing automation, we will talk about testing the interface with connection to external resources (database, file system, services, etc.).
')
On the first major project in which I participated, there were no testers. The developers, before uploading to the combat server, themselves checked the main parts of the system. With this approach, bugs are often crawled out.
It was obvious to everyone that programmers love their code and are not very enthusiastic about testing a newly written feature. About testing previously written features, and even other developers, we can’t say anything at all. The programmers themselves will not re-run all sorts of scenarios to work with the system and go into all the tabs of all pages.
In fact, on this project we could afford a similar approach. Our users sat within reach. It was a project for the internal automation of the company. Operators and managers who worked with the System could come to us and discuss the functionality, point out errors. Accordingly, we could make a quick fix and upload the revised version.
Error reports boiled down to screams behind the wall. Losses at bugs were insignificant, the situation suited everyone.
Judging by the survey that I conducted. How is the testing process arranged for you? most can afford just such an approach to testing.
We started to make a new system, which should have been used not only by the people behind the wall, but also by outside users. There is a need to hire a tester who would protect us from bad releases.
I hired a tester and, to begin with, we decided that testing should be automated, like running unit tests. Automation was supposed to save the tester from constantly repeating the same scenarios.
The idea was simple and seemed to everyone to solve problems. We wanted CI to launch an integration test suite at night. In the morning we come, if the tests are all green, it means you can make a release. If there are red tests, then we fix, run the entire test suite again, etc. until all the tests turn green.
For recording tests, we tried various options, settled on Selenium . At first, the tests were written on the Selenium API itself. Over time, a wrapper above the low-level API began to be created, and this wrapper was turned into a DSL for testing.
While the tester was dealing with the products for testing and was developing the test automation process, development did not stop. We had to release new versions after every 1-2 iterations. The tester was constantly in the stage of chasing programmers. His goal was 100% coverage of scenarios so that we can safely release new versions without fear of bugs.
Problems with test automation were unexpected for us. In fact, we were not ready for them.
The larger the system, the more tests we wrote. Tests began to work really slowly. If they took place in 14 hours - it was incredible luck. Over time, the tests no longer fit into the gap between 6:00 pm of the last day and 9:00 am of the current day. We did not receive feedback from the tests, there were downtime at work. It also happened that the lights would be turned off or the server would reboot, then the time losses were too great.
Green tests have never been. True once were, on December 30 they started and gave the result as 100% green. Perhaps it was their gift to us for the New Year. More of this all time was not.
Why they were not green? There were several reasons for this:
Due to the instability of the tests and the fact that we did not have time to update them, we had to:
We had a goal - to see green integration tests and, without fear of bugs, upload to the combat server. In fact, we have not achieved this goal.
As a result, the very first launch of the system in real conditions showed that auto tests pass a lot of bugs. We found these bugs when we manually poked the system. In 10 minutes of manual testing, the most critical bugs were discovered.
I decided to cancel the work on all integration tests and entered manual testing with writing test scripts in Google Docs. After that, we finished the main bugs and testing was perfectly integrated into the Kanban stream.
At the moment, in my company, we with a team of testers follow the approach: manual testing + writing test scripts in Google Docs. Testers manually pierce all the scripts before pouring.
Test scripts are considered part of the project artifacts and are delivered to the customer along with the source code.
In fact, it gives excellent results. In releases we have only minor bugs, or bugs, which are in fact features.
It is necessary to understand the limits of applicability of the approach to test automation. For example, if you write a bug, which catches errors 404 and 500 on a live server, then the effort is justified.
If you have a simple application that sometimes changes, it makes sense to make a set of integration tests for it, but combine it with manual testing.
If you want to replace manual testing with 100% automatic, then consider how to avoid the problems described above. Perhaps we had to first use manual testing, and then, automate those parts, the automation of which will give the maximum effect without the need to maintain these tests.
I will be glad to hear how testing has evolved in your company.
Links
Top Five (Wrong) Reasons You Don't Have Testers , Joel Spolsky
The Joel Test: 12 Steps to Better Code , Joel Spolsky
Monkey against robots. Part I (TestLabs09) , Maxim Dorofeev
Monkey against robots. Part II (TestLabs09) , Maxim Dorofeev
Monkey against robots. Part III (TestLabs09) , Maxim Dorofeev
Organizing team work in Agile: development + testing
I think it is worth mentioning at once that at all stages we used:
- Unit tests with a coating of about 50%
- Continuous Integration with the launch of unit tests (later integration tests), automatic build and release release
- The intersection of flexible methodologies under the common name ScrumbanXP
Everywhere, where I will talk about testing automation, we will talk about testing the interface with connection to external resources (database, file system, services, etc.).
')
Step 1. No testers
On the first major project in which I participated, there were no testers. The developers, before uploading to the combat server, themselves checked the main parts of the system. With this approach, bugs are often crawled out.
It was obvious to everyone that programmers love their code and are not very enthusiastic about testing a newly written feature. About testing previously written features, and even other developers, we can’t say anything at all. The programmers themselves will not re-run all sorts of scenarios to work with the system and go into all the tabs of all pages.
In fact, on this project we could afford a similar approach. Our users sat within reach. It was a project for the internal automation of the company. Operators and managers who worked with the System could come to us and discuss the functionality, point out errors. Accordingly, we could make a quick fix and upload the revised version.
Error reports boiled down to screams behind the wall. Losses at bugs were insignificant, the situation suited everyone.
Judging by the survey that I conducted. How is the testing process arranged for you? most can afford just such an approach to testing.
Step 2. Start test automation
We started to make a new system, which should have been used not only by the people behind the wall, but also by outside users. There is a need to hire a tester who would protect us from bad releases.
I hired a tester and, to begin with, we decided that testing should be automated, like running unit tests. Automation was supposed to save the tester from constantly repeating the same scenarios.
The idea was simple and seemed to everyone to solve problems. We wanted CI to launch an integration test suite at night. In the morning we come, if the tests are all green, it means you can make a release. If there are red tests, then we fix, run the entire test suite again, etc. until all the tests turn green.
For recording tests, we tried various options, settled on Selenium . At first, the tests were written on the Selenium API itself. Over time, a wrapper above the low-level API began to be created, and this wrapper was turned into a DSL for testing.
While the tester was dealing with the products for testing and was developing the test automation process, development did not stop. We had to release new versions after every 1-2 iterations. The tester was constantly in the stage of chasing programmers. His goal was 100% coverage of scenarios so that we can safely release new versions without fear of bugs.
Step 3. Fight for green tests
Problems with test automation were unexpected for us. In fact, we were not ready for them.
Long feedback
The larger the system, the more tests we wrote. Tests began to work really slowly. If they took place in 14 hours - it was incredible luck. Over time, the tests no longer fit into the gap between 6:00 pm of the last day and 9:00 am of the current day. We did not receive feedback from the tests, there were downtime at work. It also happened that the lights would be turned off or the server would reboot, then the time losses were too great.
Green tests? No, did not see
Green tests have never been. True once were, on December 30 they started and gave the result as 100% green. Perhaps it was their gift to us for the New Year. More of this all time was not.
Why they were not green? There were several reasons for this:
- Fragile tests - the interface is constantly changing and the tests did not keep pace with these changes.
- Inflexible tests - the system architecture is constantly being improved, there are generalizing solutions for code flexibility. The test code (description) also needs to be refactored and generalized to make them easier to maintain. For example, we should have many typical workflows with minor changes. Developers
coming up with an architecture that is flexible enough to allow
quickly implement these workflows. Accordingly, the test architecture should
to improve in a similar way and “keep up” with the improvement of architecture
code, which was not done due to insufficient qualifications of the tester. Of-
for the fact that the test architecture "does not have time" for the code architecture, programmers
implement the features faster and faster and the backlog of the tester accumulates - Minor bugs - tests did not always find critical bugs, sometimes they fell because of minor bugs that we knew about, but correcting them was not a priority. For this, they even specially made a plugin for TeamCity and it was possible to watch whether the test should fall or not. A set of falling tests with minor bugs did not allow the tests to be green.
- Unreadable tests - for example, we have a complicated script with a large
number of actions, the result of which you want to test. Respectively,
The test can not be divided into several small ones (not an intermediate result is checked,
and the final, which is achieved as a result of several actions). The code for such a test
need to be optimized so that it remains understandable and readable, which is also not
managed to achieve. - Long scenarios - to implement long scenarios we need to do, for example, 10-20 steps. The intermediate result is not important, only the final one is important. There are a number of problems with long scenarios:
- If an error occurred at the 2nd step, then the entire script is considered to be in error. In this case, the red test says not about the error in what it tested, but in the error of the step that was somewhere in the beginning (they could change the Login interface).
- In long scenarios, the steps are often repeated, duplication occurs.
- The long scenario cannot be started from the middle, because for this it is necessary to prepare a special state of the system, in which it must be to the N-th step. This brings additional fragility to the test structure.
Due to the instability of the tests and the fact that we did not have time to update them, we had to:
- Switch the developer to help testers switch to SpecFlow , otherwise testers did not keep the amount of tests in C # code. The head of the testing department at that time gave a report on the subject of acceptance tests on a cucumber
- 70% of the time testers rewrote automatic tests to match the current set of system functions. Those. it was a job for the sake of the tests themselves.
- As a consequence of the above, we hired more testers.
Is the goal achieved?
We had a goal - to see green integration tests and, without fear of bugs, upload to the combat server. In fact, we have not achieved this goal.
As a result, the very first launch of the system in real conditions showed that auto tests pass a lot of bugs. We found these bugs when we manually poked the system. In 10 minutes of manual testing, the most critical bugs were discovered.
I decided to cancel the work on all integration tests and entered manual testing with writing test scripts in Google Docs. After that, we finished the main bugs and testing was perfectly integrated into the Kanban stream.
Current state
At the moment, in my company, we with a team of testers follow the approach: manual testing + writing test scripts in Google Docs. Testers manually pierce all the scripts before pouring.
Test scripts are considered part of the project artifacts and are delivered to the customer along with the source code.
In fact, it gives excellent results. In releases we have only minor bugs, or bugs, which are in fact features.
Against test automation?
It is necessary to understand the limits of applicability of the approach to test automation. For example, if you write a bug, which catches errors 404 and 500 on a live server, then the effort is justified.
If you have a simple application that sometimes changes, it makes sense to make a set of integration tests for it, but combine it with manual testing.
If you want to replace manual testing with 100% automatic, then consider how to avoid the problems described above. Perhaps we had to first use manual testing, and then, automate those parts, the automation of which will give the maximum effect without the need to maintain these tests.
I will be glad to hear how testing has evolved in your company.
Links
Top Five (Wrong) Reasons You Don't Have Testers , Joel Spolsky
The Joel Test: 12 Steps to Better Code , Joel Spolsky
Monkey against robots. Part I (TestLabs09) , Maxim Dorofeev
Monkey against robots. Part II (TestLabs09) , Maxim Dorofeev
Monkey against robots. Part III (TestLabs09) , Maxim Dorofeev
Organizing team work in Agile: development + testing
Source: https://habr.com/ru/post/145974/
All Articles