And again in our tape "Calendar tester . " This month Marina Tretyakov, a tester of the Kontur project. Deliveries , will tell about test optimization. Marina will analyze specific problems and ways to solve them, and also advise how to optimize your tests and reduce the time for testing.

First of all, let's divide all the testing according to the degree of automation in order to examine each type in the future in more detail.

According to the degree of automation tests are divided into:

1. Hand held
2. Automated.
3. Automatic (without human intervention, at the moment - rather a myth than a reality).

The approach to test optimization directly depends on the degree of their automation.

Optimizations applicable to all types of tests

Testing includes the steps:

• preparation of the tested system,
• preparation of input data
• carrying out tests (manually or automatically, consider below),
• collection and analysis of results.

It is believed that manual regression takes the most time for testers. However, in most cases this is not the case. At a minimum, it is not worth asserting it until the problem is measured and proven.

Consider solving common problems. Many tips may seem very obvious to you, but as the experience of conferences and presentations of colleagues from other companies shows, these tips are still relevant and have proven their applicability and usefulness.

Problems:

1. Long training of the system under test.

Questions that are important to ask before the introduction of optimizations:

• Long regarding what?
• Who is involved in this training (testers, programmers ...)?
• How many times can you prepare a system under test in a working day? Does this number match testing needs?
• What is the longest preparation stage and why?

To find the cause of this problem, it is important to ask the right question.
Consider examples:

Long compilation of all system modules
The correct question is : do you need to compile all the modules?
Solution : to compile not all modules of the system, but only those that were affected in the task, and which will participate in the release.

A long manual process of updating the entire system for different "virtual" on the test bench
The correct question is : at what point in the update is a person required to participate, and in which not?
Solution : to automate the layout process, use special deployment tools and rolling out services or debugged release mechanisms for “combat”, but use them only for deploying to a test one.

The long process of "spilling" the source for the "virtual" on the test bench for further compilation and rolling
Probable problem : network interaction.
The correct question is : long about what (gathering on a local machine, gathering on a local network)?
Solution : the test bench and the place where the sources are located must be in the same network in order to minimize network interaction.

I encountered this problem when I decided to change the test site in Yekaterinburg to Moscow. And in the process of testing the “layout” of the site, we quickly noticed that the stand update did not take 3 minutes, but almost 15 minutes. The reason was that the source code with a large number of small files was located in Yekaterinburg, and the stand in Moscow. The layout process “rested” on the network transfer of small files for further compilation and “layout” on the stand. As a result, the code also “went” to Moscow :)

2. Long collection and analysis of results

Questions that are important to ask before the introduction of optimizations:

• Long about what?
• What are the stages of the process of collecting and analyzing the results? What is the longest stage and why?
• Who analyzes the results?
• Who decides on the release and on what basis? How much time is spent making a decision?

For example :
The test results are drawn up on the template, the design on the template takes most of the time when testing.

Solution (thanks, Cap!) : Refuse to fill in the results by the template or create a template that is easier to fill. It will be necessary to agree with the team and find out from those who read these results (do they read them?), Is there really a need for just such a template (the risk is to write test results “in the table”).

Optimizations applicable to manual tests

These tests can be divided into two large classes:

1. conducted regularly, for example, before release (count regression testing),
2. conducted rarely and only to test new functionality.

If the regression is carried out frequently and manually, then it makes sense to think about automation and automation payback, but in this article we will not consider automation ROI.

For manual tests (not regression tests), we should speak not about test automation , but about instrumental support for testing (as advised by Alexey Barantsev at the training, which he conducted in our company). In this case, autotests will act as a tool. In this context, the logic and view on autotests in general will change.

When testing manually, first of all, it is necessary to look for routine tasks (tasks, not tests!), And to optimize them (using automation or redistribution of human resources).

For example, the routine task for tests is to prepare test data. There are different ways to do this training:

1. manually via UI,
2. manually through the API,
3. by running autotests, then the data will be a side effect of these tests,
4. automated via scripts / utilities / self-written tools via API or UI.

If you have never wondered how much time it takes to manually prepare test data, can it be time to measure it? And it turns out that it is much more efficient to use at least the second, and preferably the 3rd and 4th approaches.

Optimizations applicable to automated tests

The problem of preparing test data here is more acute than with manual testing. Preparation of test data should be:

1. quick,
2. resistant to design / layout changes,
3. resistant to possible parallel runs of tests,
4. resistant to changes in the internal system architecture.

It is desirable that data preparation does not require additional skills and time to implement the solution.

Automated test data can be prepared:

1. via UI,
2. via API or HTTP requests,
3. through database queries.

Consider the pros and cons of these approaches in more detail in the table:

Preparation of test data via API or HTTP requests for a combination of pros and cons is the most optimal.

There are a number of the most common optimizations applicable to automated tests:

Test parallelism

If one of the problems of tests is precisely the time they pass, while there are computing resources, you can parallelize and execute them in one of three parallelism modes:

1. Parallelism on one computer, parallelism on processor threads.
2. Parallelism on different computers.
3. The combination of the first and second methods, that is, if there are several computer machines, the tests pass in parallel along the threads on each and in parallel among all the machines.

Delete old tests

If tests are written, they pass, but they don’t really check anything (for example, there was business logic before, now it’s not and tests don’t check anything), then such tests should be ruthlessly deleted, because in fact they don’t carry any meaning taking the extra time from the run. It is also worth removing the tests, the result of passing which does not affect the decision about the possibility of release.

Application of test design techniques to optimize test case sets

In manual testing, various test design techniques are required to optimize test case kits. For autotests, you should also use the method of equivalence class breakdown, pairwise, boundary analysis and many other techniques to optimize the set of autotests.

Transferring existing tests and checks to another level

For example, there is a browser test that opens a search line, enters “apple”, “apples”, “apple”, “apples” (and so on), and looks that when searching for it, it seemed to him that a notification about buying apples in the store (test looks at the very fact of displaying the notification and no more). Thus, a long test for UI does not in its essence check the UI, it checks the logic that the unit test can test, so this test should be removed and a unit test written instead.

The correct decomposition of tests on the levels of “modular - integration - system”

I will give an example. There is a manual script: select a product in the online store, put it in the basket and proceed to checkout. What can be done (and it will be wrong): create exactly one test that will look for the product, add it to the cart and proceed to the checkout.

In this case, it would be correct to first divide the test into three sub-scenarios: product selection, product addition to the basket, order placement. Each scenario is divided into more atomic checks.
For example: “store opening - displaying different categories of goods for selection” - one test; “Selecting categories from different product categories” is another test. We consider each test in more detail and determine what level of tests are needed for it, the previous example can suggest which type of tests it is better to immediately issue as modular.

The popular connection scheme of the test and testing systems for automated testing of web applications:

To optimize automated testing of web applications, it is advisable to consider optimizing each interaction in the described scheme.

For simplicity, consider optimizing some of the interactions:

1) Interaction “test cases - browser - database”
Using the API not only to prepare the data for the test, but also to carry out a number of steps in the test.

For example, if the goal is to check the UI at the end of a long chain of actions, then it is not at all necessary to carry out all actions through the UI. After all, if in the middle of the chain in the UI something is broken, the test will not reach the end and the target check. The tester will guess so, and if they fix this broken link in the chain, then everything that works after it? If in this case, throughout the whole chain except for the last action, to use the API, then with a UI breakdown of any link, the tester will know whether the system will work as intended if the developers fix the broken link.

2) Interaction “test-cases - SeleniumWebDriver - browser”.

• Closing extra tabs at the end of the test, instead of closing the browser.
  On my project, this optimization helped save 10 minutes in the UI test run (instead of 1 h. 10 min., Tests began to pass in 1 h.). This optimization is connected with the logic of SeleniumWebDriver, which is used on the project - it has a very long preparation for opening the browser, but closing the tabs happens almost instantly.
• Optimization of the application cache of the system under test so that the tests run faster.
• Use headless browsers to avoid the cost of rendering elements of web pages.

Finally

For any optimization, it is necessary to clearly define for yourself the current problems in the testing process, break down the items in what they are, present possible options (better a few!) To solve them. After that, it is required to voice them in a team, “sell” your ideas and proposals for a solution, and only after general approval can you distribute the efforts and solve the set tasks. The preliminary “Before” rating and the “After” assessment will help to consider all the benefits from process optimization.

And once again I would like to repeat: do not look for routine tests, look for routine tasks, and automate them!

List of calendar articles:
Try a different approach
Reasonable pair testing
Feedback: as it happens
Optimize tests
Read the book
Analytics testing
The tester must catch the bug, read Kaner and organize a move.
Load service
Metrics in QA service
Test security
Know your customer
Disassemble backlog