Test Automation: Cons
Autotest (automated test) is a script that simulates user interaction with an application, the purpose of which is to localize software errors.
The ultimate goal of test automation is a set of autotests, which, when you press the “GO!” Button, will be launched one by one. Well, or you can always run any autotest separately, if there is a need for it. Each such script checks the correct operation of a certain part of the application and fixes errors in case something is wrong.
About how much more profitable to use automated testing than manual, as well as about the advantages of automated testing is well written in this article. I want to describe the possible problems that may be faced by the tester, who decided to use automated tests.
Problems arise when choosing test cases that need to be automated. Not all test cases can be automated, and there are situations where some of them are best left to manual testing. In other words, if you have 1000 test scenarios, then there is very little chance that 1000 of them can be automated. It is rarely possible to switch completely from manual testing to auto tests. Do not forget that manual and automated testing is not mutually exclusive, but complementary methods.
For example, it is impossible (read: very difficult) to automate testing of such things as:
Not always it turns out that the development and maintenance of automated tests will be cheaper than manual testing. It is necessary to spend money on the development environment, system administrators, hiring programmers, their information support (meaning support if you have questions about the system under test and test cases), to provide them with jobs. All this will result in a considerable amount and will pay off only when testing large systems, when it is possible to automate a fairly large number of tests. After developing the scripts they need someone to run and analyze the results. It will not be possible to conduct testing according to the “launched — waited — received the final result” scheme. We need a person who can analyze the performance of autotests: determine whether to send a test for revision, if there is an error found by an autotest, to prepare data for testing. On the preparation of the development environment, the development itself will take a lot of time.
')
A separate item on the cost of automation is autotest support.
Auto tests always require support. Since autotest writing is programming, in order to change \ correct the autotest test logic you need to go into the source code. To do this, you need to hire additional specialists who will accompany the written scripts. These are new expenses, without which one cannot do and which significantly increase the cost of autotests. There are 2 main cases in which you may need to intervene in the source code:
It is good if during the execution of several iterations of testing an application you can always submit the same data to the autotests for input. For example, checking the entry in the database, or checking the opening of a menu where input data is not required at all.
But it also happens that before each launch of the autotest data must be prepared anew. With a large number of tests, the preparation of input data can take a long time. For example, when testing the banking system may require accounts in rubles, dollars, euros; accounts with different amounts owned both by individuals and organizations. After running a series of autotests, a zero balance may remain on all accounts, some of them will be closed, some will be blocked. Subsequent testing will need to use other accounts. In some cases, the data may be hardcoded in autotest scripts, and this may not happen because the developer was too lazy to make his script more flexible, but simply because a test case required such a hardcode. After some time, the test case will change (for example, the name of the bank will change from “Habrabank” to “Habrakreditbank”), and a script change will be required.
Especially important for regression testing - testing new versions. At each update of the interface or functionality of the software being tested, the autotest will need to be improved. Auto tests cannot themselves adapt to the new interface, and to continue their correct work, you have to change everything manually. The greater the number of tests, and the more innovations occurred in the software, the longer it will take to update.
Example: in the old version of the program there was a menu “File -> Options”, in the new version this item was divided into two: “File -> Settings” and “File -> Parameters”. The autotest will still look for the “Options” item, and even if you are cracked, it will not find the “Settings”.
Or it may change, for example, the loading time of the test page. If earlier it was loaded instantly, then now it takes a minute. Suppose now the connection speed is weak, or the base slows down. The autotest does not know this, waiting for the page to load for 10-15 seconds and displays the message “Error: page did not load.” Again, you need to refine the source code of the script.
In both cases, much depends on the developers: the process of writing autotests requires no less knowledge and effort than the process of creating any other software. Consequently, there are all the problems associated with the development: the complexity of quality and reliability control, postponement, budget changes, and many others.
These cons do not mean that you should avoid auto-tests at all costs. They also have benefits.
As can be seen from the graph, the use of automation gives the best results at the same cost. But, once again, these results will be best only when automating the testing of large projects.
A few more advantages of automation:
The ultimate goal of test automation is a set of autotests, which, when you press the “GO!” Button, will be launched one by one. Well, or you can always run any autotest separately, if there is a need for it. Each such script checks the correct operation of a certain part of the application and fixes errors in case something is wrong.
About how much more profitable to use automated testing than manual, as well as about the advantages of automated testing is well written in this article. I want to describe the possible problems that may be faced by the tester, who decided to use automated tests.
Selection of tests for automation
Problems arise when choosing test cases that need to be automated. Not all test cases can be automated, and there are situations where some of them are best left to manual testing. In other words, if you have 1000 test scenarios, then there is very little chance that 1000 of them can be automated. It is rarely possible to switch completely from manual testing to auto tests. Do not forget that manual and automated testing is not mutually exclusive, but complementary methods.
For example, it is impossible (read: very difficult) to automate testing of such things as:
- installation of the operating system
- printer printed document check
- checking the contents of the picture, video
Cost of
Not always it turns out that the development and maintenance of automated tests will be cheaper than manual testing. It is necessary to spend money on the development environment, system administrators, hiring programmers, their information support (meaning support if you have questions about the system under test and test cases), to provide them with jobs. All this will result in a considerable amount and will pay off only when testing large systems, when it is possible to automate a fairly large number of tests. After developing the scripts they need someone to run and analyze the results. It will not be possible to conduct testing according to the “launched — waited — received the final result” scheme. We need a person who can analyze the performance of autotests: determine whether to send a test for revision, if there is an error found by an autotest, to prepare data for testing. On the preparation of the development environment, the development itself will take a lot of time.
')
A separate item on the cost of automation is autotest support.
Support
Auto tests always require support. Since autotest writing is programming, in order to change \ correct the autotest test logic you need to go into the source code. To do this, you need to hire additional specialists who will accompany the written scripts. These are new expenses, without which one cannot do and which significantly increase the cost of autotests. There are 2 main cases in which you may need to intervene in the source code:
1. Change the input to the test
It is good if during the execution of several iterations of testing an application you can always submit the same data to the autotests for input. For example, checking the entry in the database, or checking the opening of a menu where input data is not required at all.
But it also happens that before each launch of the autotest data must be prepared anew. With a large number of tests, the preparation of input data can take a long time. For example, when testing the banking system may require accounts in rubles, dollars, euros; accounts with different amounts owned both by individuals and organizations. After running a series of autotests, a zero balance may remain on all accounts, some of them will be closed, some will be blocked. Subsequent testing will need to use other accounts. In some cases, the data may be hardcoded in autotest scripts, and this may not happen because the developer was too lazy to make his script more flexible, but simply because a test case required such a hardcode. After some time, the test case will change (for example, the name of the bank will change from “Habrabank” to “Habrakreditbank”), and a script change will be required.
2. Changing the functionality
Especially important for regression testing - testing new versions. At each update of the interface or functionality of the software being tested, the autotest will need to be improved. Auto tests cannot themselves adapt to the new interface, and to continue their correct work, you have to change everything manually. The greater the number of tests, and the more innovations occurred in the software, the longer it will take to update.
Example: in the old version of the program there was a menu “File -> Options”, in the new version this item was divided into two: “File -> Settings” and “File -> Parameters”. The autotest will still look for the “Options” item, and even if you are cracked, it will not find the “Settings”.
Or it may change, for example, the loading time of the test page. If earlier it was loaded instantly, then now it takes a minute. Suppose now the connection speed is weak, or the base slows down. The autotest does not know this, waiting for the page to load for 10-15 seconds and displays the message “Error: page did not load.” Again, you need to refine the source code of the script.
In both cases, much depends on the developers: the process of writing autotests requires no less knowledge and effort than the process of creating any other software. Consequently, there are all the problems associated with the development: the complexity of quality and reliability control, postponement, budget changes, and many others.
And still
These cons do not mean that you should avoid auto-tests at all costs. They also have benefits.
As can be seen from the graph, the use of automation gives the best results at the same cost. But, once again, these results will be best only when automating the testing of large projects.
A few more advantages of automation:
- Auto tests run faster than people
- Autotest performed with greater accuracy
- Test automation allows you to improve the quality of the product.
- Automation can be used in almost all testing processes.
- Auto tests can be performed at night.
Source: https://habr.com/ru/post/111292/
All Articles