Test documentation. Turn tables into trees

In the previous article I described how the first stage of project testing is going on in our company - analysis. Today I will talk about the next stage - designing and documenting tests.

This stage is optional. Some projects do not have documented requirements, and then often supporting test documentation is the only sensible way to store and transfer knowledge about the product. Sometimes the test documentation is required by the customer, sometimes we write it for ourselves. Sometimes, if we have well-written requirements, we refuse to document tests in order to save resources.
')
The type of test documentation also depends on the situation on the project and the expectations of the customer.

Check lists vs Test cases

The checklist differs from the test case by the degree of detail. In the checklist, you will not meet the detailed steps of the case, to use the checklist when testing a lot of information you need to keep in mind when running the tests and know the logic of the application to work perfectly.

Our company has always used checklists, because it takes an unreasonable amount of time to write test cases, and they are too heavy — reading a case and understanding it also takes time. In addition, do not forget about the effect of the pesticide - code bugs tend to adapt to the tests. When using a checklist, some freedom of action is retained, and test cases are completely deprived of this freedom, increasing the effect mentioned above. However, when the checklist is run for the seventh time in the last day before the release, some of the functionality laid down under one item of the checklist is lost due to the human factor.

It was decided to expand the checklists and make them more detailed. So the tester, unconsciously chasing the feature before the release, will not forget to check the network errors in response to each request, will not lose checking any "unimportant" button or any one of the twelve. So we came to writing a detailed user story, completely covering the application feature, but in fact being one huge test case.

Advantages of this approach:

• checklist implies continuous execution by the tester. Accordingly, all cases are arranged in a convenient manner for the run, and no time is spent on moving to the next case.
• checklist covers a large number of user scripts
• The checklist contains both positive and negative cases. Checks for error and interrupt recovery, which is very important for mobile applications.
• checklist means long sessions, which increases the likelihood of detecting memory leaks and navigation problems
• information about requirements is received by the tester consistently, which gives a better understanding of the logic of the application

Cons of this approach:

• The checklist implies a decomposition on the application screens and is strongly tied to the design. Accordingly, all changes in the design imply changes in the checklist
• taking into account that such check-list cases should be connected in one story, and there can be up to two hundred cases in one check-list, it is difficult to maintain
• the effect of the pesticide is greatly enhanced
• checklist cannot replace requirements for developers - it’s inconvenient to develop for such requirements
• the checklist is long enough to write and because of its cumbersome it is not always convenient to use - in fact it comes in handy only in the final before the release

For a long time we lived like this. Led their checklists in the test management system Sitechko. But the company was changing, business processes were changing, and the question of switching to another TMS was in the air. He became the impetus for the revision of the process of maintaining test documentation.

Tables vs Trees

Once I drank multivitamins, and it dawned on me that instead of storing tests in tabular formats, it’s more convenient to use trees, or rather, nested lists. After all, when writing the very largest user story, we often face the problem of not knowing where to place alternative actions. For example, when we have several buttons on the alert. To check each of them, we have to register the call of this alert several times. Instead, we could suspend checking each of the buttons for an alert call.

In general, the idea was to write the same user scenarios in the form of a tree in which the transition is an action, and the node is the state in which the application is after this action. In fact, the state transition diagram, only the objects are the application screens. Each branch of such a tree is a test case.



When they began to try, they ran into problems. It turned out that the usual decomposition on the screens of the application does not work: it is inconvenient to rely on the design when designing tests. Tree branches grew far into the depths, and it was uncomfortable visually. In pursuit of the script, we turned back the cycles. And it also became clear that it is impossible to refuse the tables.

The decision was in the change of approach to decomposition, greater awareness and rejection of the “default solutions”. The tree structure of the test documentation is really convenient, because it gives more freedom in the design. The decomposition view determines what exactly will become the nodes of our tree. And this in turn is determined by the features of the product and customer priorities.

In general, the advantages of using a tree structure:

• As a result, the structure of such documentation is very flexible and allows you to maintain both checklists and test cases, depending on the needs of the project.
• the tree is represented as nested lists: the nodes of the tree have a certain order, which preserves the possibility of a consistent and structured transfer of information about the requirements to the tester if there is no documented TK on the project
• This structure allows you to design test documentation that can be transferred to developers instead of TK
• time spent on writing checklists and test cases is reduced, because the structure allows you to avoid copy-paste

Cons of the approach:

• such a structure of test documentation requires careful design and preliminary analytics - with poor design, all the advantages mentioned above are lost

Summary Patterns

Application screens

The source of knowledge is navshema. The first level of the tree is a list of case studies, which usually corresponds to sections of the application. Next, they are hung with a list of screens of each section, to each screen - a list of its states. Each node of the tree, starting with the third level, may contain a checklist in tabular format describing each design element and how to interact with it. If the design elements are complex and have many states or there are duplicate elements on the screen, you can decompose even deeper. Thus, one branch of the tree describes the life cycle of a single screen.

Below, as an example, the general scheme of reasoning is given when decomposing the order section of the ticket aggregator.





We fasten short checklists to the leaves of this tree. So to each sheet of the “navbar” link the checklist for the navbar elements for the current screen:



And for each sheet of the “scheduled trips section” we link the checklist to check the part of the list with active orders:



The criteria for choosing such a pattern are as follows:

• UI is a priority for the customer
• minimum business logic on the client
• for the application are characterized by custom design elements and animations, complex gestures
• there are no other documented requirements on the project other than navshema

Objects / Actions

With this approach, we focus not on the whole scheme, but on the documentation of the IPA and the client's business logic. Negative and positive cases are divided into different branches. It is desirable that elements of the same tree level answer one question, but you can leave this restriction only for one level of the hierarchy.

Such a scheme is extremely convenient in cases where we have an active influence of users on each other, which gives rise to complex scenario chains. An example is the chat. With respect to the previous approach, such documentation is easier to maintain, since changes in logic happen less frequently than in design.

Below is an example of a general scheme of reasoning for decomposition on the principle object / action.





In this scheme, an additional bonus is the ability to use it as a map for exploratory testing and for Smoke test. The degree of testing details can be adjusted by cutting off the tree levels during the run, as each next level specifies the previous one. When you go deep into a tree branch, you go deep into functionality.

For example, for the chat already mentioned, the documentation scheme would look something like this:



Criteria for choosing such a pattern:

• the goal is to test the functionality
• complex business logic
• frequent design changes
• testing strategy on a project involves a combination of scripting and research testing

Based on use cases

There are situations in which it is unprofitable to decompose the functionality and design tests according to two previously described schemes. For example, if we want to cover long-term work with an application with tests, as is the case with a social network tape or listening to music in the background. Or when the feature is tied to scripts with a small number of alternatives - for example, subscribing to a content. In this case, we use the third pattern based on user scripts.

First we decompose the functionality of use cases. We decide on which actors can participate in the process of working with the application and what goals they can set for themselves. The first two levels of our tree will be responsible for this. Next, we try to find all possible input conditions that may affect the development of the scenario to achieve the current goal, and structure them in the tree. They are also the most convenient to divide into positive and negative. Next to each sheet we suspend the scenario check list to check the functionality responsible for achieving the goal.

As an example, below is a diagram for a music player with the function of downloading tracks for listening offline:





Here, to all the leaves of the positive scenario, we suspend the checklist, which will need to be run under different network connections:



It so happens that when thinking about possible use-cases, the goals of users turn out to be very global. For example, in the already mentioned air ticket aggregator, the goal “to buy a ticket” can stump with an abundance of possible preconditions and the number of steps that must be taken to achieve the goal. In addition, in such an application, much depends on the behavior of third-party systems, which imposes some restrictions on the definition of all preconditions and the uniquely executed script. Offers come from different airlines and can change at any time. At any given time, it is impossible to guarantee that the purchase of the ticket will be successful, since this ticket may be purchased while we are filling in the reservation data.

The solution to the first problem is a more detailed decomposition. That is, the big goal “to buy a ticket” can be broken down into small goals, corresponding to the clearance steps - “read the offers”, “fill in the passengers data”, “pay for the order”. And then find a set of possible preconditions, user actions and results for these small goals.

The solution to the second problem is less obvious. This is generally a limitation of the use case - in case if the behavior of the system is not uniquely determined by the user's actions, there are problems with the coverage and design of use cases. For ourselves, we decided that in such situations we try to prescribe all possible behaviors of systems not subject to the user, as preconditions, and thereby reduce the uncertainty of the result of the script. Or use a different design scheme for test documentation.

Criteria for choosing such a pattern:

• priority for the customer is the correct operation of certain user scenarios
• The application contains screens where the user will spend a lot of time consuming content.
• The application contains features that consist in working out a linear script.
• testing strategy on a project involves scripting
• simple business logic easily covered by use cases

The patterns described above can and should be combined. In a large application with high quality requirements, there are sure to meet both features that are optimally broken down by object / action, and features that require a scenario description of use case. And in the same way, in such an application, screens can be found for which it is necessary to prescribe all the states of the elements and all sorts of user interactions with them. Therefore, it is not necessary to choose one single documentation scheme for the entire application as a whole, it is better to always look at the circumstances.

Pushing away from the goal

We changed the tool and developed new approaches to test documentation, but did not reject the old approaches. The choice of strategy depends on the needs of the project and the priorities of the customer. If the application logic is simple and the project does not last long, then standard checklists for functionality with minimal detail are usually sufficient. If the project is large, complex and without requirements, then part of the features should write a full-fledged "tree" documentation. If the project has well-documented requirements, sometimes you can not waste time writing tests on functionality, but you can pay more attention to non-functional testing (performance, security) and systematize it - again, if there is an appropriate agreement with the customer. Or document only "risky" tests. User-story is still almost always written, but not as detailed as the acceptance for the customer or as a smoke test, and the decomposition work carried out before this helps us to quickly design the script and set the priorities correctly.

The availability of test documentation on a project allows you to capture information about requirements, plan and structure tests in advance, lower the threshold for a new tester to enter a project, and reduce the risk of missing errors due to human factors. However, writing and maintaining test documentation requires resources that are not always possible or not always worth spending.