The brilliance and poverty of test automation

It is believed that the presence of automatic tests - this is an absolute good. If developers write tests, that's good; the more tests, the better. At the same time, in reality they are not often written, and all testing is done manually.

Do not blame this state of affairs on incompetence, stupidity, or a banal laziness of developers. Compared to manual testing, automated has both advantages and obvious disadvantages. If there were only advantages, and there would be nothing to talk about.

Benefits of Auto Tests

Automated testing has several significant advantages compared with manual testing.
')
Cheapness - running automatic tests is always cheaper than manual testing by several orders of magnitude. Development of automatic tests is certainly more expensive than one - time manual testing.

Indefatigability - automatic tests can be run again and again, they have an unlimited working day, they do not get tired, and do not get sick.

Speed - automatic tests run orders of magnitude faster than the same manual checks. Of course, there are relatively slow tests, but this is rather an exception, confirming the rule.

Accuracy - automated tests check exactly what is described in the script, in all its details and trivia. They are not distracted, not confused and not forgotten.

Comprehensive - automated tests can cover a huge number of scenarios and find induced errors in such far corners of the application that the manual tester would never have reached except by chance.

Benefits of manual testing

Manual testing, however, is superior to automated in many aspects.

Agility - check something manually for the first time easily and quickly. An automatic test must first be developed. Almost always it is much slower than manual checking.

Flexibility - manual testing can be carried out much more diversely, and changing the testing method costs almost nothing. Let's test in Safari - please, on a Chromebook - no problem, IE6 - you have to start the virtual machine, but you can, too.

Adaptability - the ability to quickly adapt to change. When a product changes dramatically, and things start to work at all differently from before, manual testers can easily forget how they tested before. They will simply test what is now. Automatic tests in the same situation will give out thousands of errors, and they will have to be corrected before moving on.

Creativity - manual testing allows you to find problems that were not known in advance, moreover, they were not even implied.

Meaningfulness - although every single thing can be absolutely correct, people understand much more easily that together these things have no meaning.

Expressiveness - it’s not enough to find a problem at the “nothing works!” Level, it is important to correctly explain what it is and be able to answer additional questions from the developer. And in this, people are also better than automatic tests.

Authenticity - most of the applications we write for people, and that is why people are best suited to test them.

Brilliance of testing paradoxes

Speaking about testing, and even more about full testing, we are a little cunning. Obviously, it 's impossible to verify everything .

All we can do in practice is to carry out some sufficient testing . As a result, we can with a certain degree of probability say that the majority of users will not encounter problems.

Having spent for testing 1 day, we will achieve 90% of satisfied users. Testing the week - 99%, testing the month 99.5%. We are spending more and more time to achieve less and less results. This is impractical.

By correcting problems quickly, you can achieve a much higher quality product than by investing all available resources in achieving full test coverage. Salt is to find a balance in the application of forces.

Manual testing gives much more return if it is necessary to develop quickly , especially if the application changes incrementally, in local predictable parts. The reason is that it’s much faster to test the change than to automate the same test.

Automatic testing gives very little return at short distances . At every single moment, deciding what is more profitable, whether to write tests or not to write, the second is always preferable.

The situation is fundamentally changing in the long term. The application code must be refactored so that it does not die. When programmers have redone everything - testers should check everything. Full cycle of regression tests. A week? Two? Month? Or worse: programmers have updated the version of the key library. The day of work, everything seems to be compiled - great! Still, testers should check everything. Madness. After going through this a few times, anyone will be ready to go for anything, just to avoid such situations in the future. To ensure a stable base level of quality, it is necessary to invest in test automation.

Unfortunately, automated tests are not always the solution, sometimes they themselves become a problem.

Poverty testing automation

Problem : you need to be able to check the application completely and quickly, without losing anything, many times in a row.

Solution : write automatic tests that do all this.

But not everything is so simple.

Writing tests is a development that is exactly the same as developing a core business logic. It requires resources and qualifications. Providing a high-quality coverage of the application code with tests, we simultaneously slow down the development as such. And we slow down not by 10%, but twice, three times or even more.

Speed ​​is a relative concept . Each particular test is, of course, performed very quickly, especially pure unit tests, when all checks are performed locally. Integration tests, REST tests are also relatively fast. What is 100 milliseconds, well, even if 1 second, even tens of seconds for selenium tests, is nonsense compared to manual testing. But when the number of tests in any significant way, the fast tests turn out to be very slow. Run tests in 5 minutes? Half an hour? Three hours, three o'clock? Two days? Even without having achieved a complete coverage of business logic, there is a serious question about how not to run all the tests, or not to run the tests every time - otherwise automatic tests start to slow down the development.

Note : Tests can be significantly accelerated by investing in hardware — allocating a separate server for them, 10 servers, 50, etc. Some companies can afford 1000 test servers, but not all.

The toolkit is bad . None of the mainstream programming platforms was designed to provide the ability to easily and fully develop tests. In Java, all test frameworks are third-party libraries. Running tests, performing checks, creating mock objects - all this is there, but in parts and outside the platform. In Go, the test framework has already been added to the language ecosystem initially, but this is only the start of tests. Choosing the right toolkit for writing tests is a separate problem that needs to be solved.

They are not so good . Tests - the same code written by the same developers. They also contain errors and all the same problems as any other code. It happens that the test does not check anything. It happens that the tests also need to refactor. It happens that in a fallen test it is very difficult to understand what the problem is. It happens that tests fall just like that, when there really is no error.

Block changes . By their nature, tests are designed to verify that nothing has changed. If something starts to work wrong - this is a bug. But when business logic changes, even if the change correctly and fully meets the new requirements, the tests know nothing about it. Changes to several rules can lead to the need to correct dozens, if not hundreds of tests, especially if the coverage is really good.

Limit refactoring . Tests also need to be refactored, even if the business logic does not change, and only cosmetic changes occur in the code. This may lead to the fact that you need to fix 5 places in the main code and another 80 in the tests. Everyone in this situation will think: is it worth doing something more than renaming the method.

Sisyphean work - this is what development through testing turns into, when tests are written before writing code, or at least along the way. Any competent developer refactor code in the process of writing it as he more and more deeply understands the problem and understands how to implement the solution in the best way. But if he writes tests at the same time as the code, in addition to changes in the code, he also has to change the tests. And unlike the refactoring of the finished code, the refactoring of the code in the work is usually much larger, which means that the required changes in the tests will be appropriate. And so - many times before the code goes to the review. Half the time in the garbage is easy.

The number of tests is deceptive : 3,000 tests in total — for the web interface, REST API, business logic, and unit tests — can be tested with a stretch of 1000 situations. All the rest is duplication.

Automatic regression does not cancel the manual contrary to the main idea. All because automatic tests check a lot of things, but not all. But no one knows what exactly they are testing and what is not. We need to check the registration. And we have 18 927 tests. If one of them is red - “everything is fine”, you can return the task to the developers - let them understand. If all tests are green - it does not mean anything until the person responsible for manual testing is sure that the registration logic is checked automatically, and that it still needs to be checked manually. And almost twenty thousand tests are impossible to understand. The result - you need to check everything manually.

It is not surprising that some people leave the development to sell plastic windows, train billiards or make coffee (and these are all real cases).

What conclusions we made in Wrike from our own and others' mistakes will be discussed in the next post.

Dmitry Mamonov

Development department,
Merge's division into master,
Department of work with git,
Leading operator bash console 1 discharge