What tests do you need? Part 2. Test matrix

annotation

In the first part of a series of articles, I talked about what the choice of tests depends on. Having in mind the understanding of what you want to achieve with testing, you can take the next step - choose tests. For this you need to understand what tests are in general.

Almost all articles devoted to types of testing, have a grouping of tests in some categories. Only this division does not always coincide. Reading into such articles, you often find discrepancies in terminology among different authors.

I set myself the goal to sort through, visualize a variety of tests to help those who choose which tests they need and those who study the testing itself.
')
I do not undertake to give clear definitions of the types of testing in this article. The interpretation of the terms used for the name of the species will be discussed in the third part. This division is associated with the amount of information.

Classification of types of testing

At the types of tests can be viewed from different sides. And classify them as difficult as, for example, culinary dishes. You can group dishes according to national cuisine, you can by reception time (for breakfast, for lunch, for dinner), you can by ingredients used, you can by calorie and so on. The products themselves - can also be divided into groups according to various criteria. At the same time, everyone understands that a steak can be an evening meal, while it does not cease to be a meat dish and cause controversy about belonging to the national cuisine. So tests - regression test can be an integration.

Belonging to one category does not exclude belonging to another.

Another example - fatty food can be called both a dish and a product - cheburek and vegetable oil. A vulnerability test can be a set of automated scripts and one test case in a set of functional acceptance tests of the input form.

A test type is a characteristic that can be possessed by both a single test script and a whole collection of tests.

See what kind of cute minds with wine:

It would seem that it completely covers all types of wines. And now look at it again and list the wines by country of production, as they are placed in the supermarket or in the restaurant menu. Does not work? How, then, to choose? This card is not suitable for selection by manufacturers. If you look closely - there is a postscript "in style and taste."

That's the same with the types of tests.

By limiting ourselves to one criterion for grouping, we lose sight of the whole variety of tests.

A grouping criterion may be a key selection criterion.

Test Type Map

I like the approach that was used in the Wikipedia article about software testing , since a lot of cuts were considered.

I took this list as a basis and supplemented it with information drawn from other sources and personal experience and compiled the mind map (knowledge map). This map is useful to me in the process of planning tests - I check if I have forgotten what. I share the scheme with you:



View full size.
Download the source in xmind format here .

Test titles are in English to avoid translation inaccuracies.

Flower Growers

• Blue highlights the features applicable to test sets.
• Yellow highlights characteristics applicable only to test cases.
• Green highlights universal specifications.

This division is conditional, there may be exceptions.

About yellow blocks in the knowledge map

Looking at my map, you might think that the division of tests according to the types of scenarios is not so important. In fact, specific test cases many times more than sets, and

having missed some kind, you lose a bunch of valuable checks, and, therefore, defects.

The types of tests that are listed in the Scripts node are actually the names of the scripting techniques. Compliance with these techniques gives specific tests at the output. Therefore, in my opinion, it is appropriate to present them as types of tests.

The mentioned techniques are considered to be the types of black box tests. Why these tests on my scheme do not apply to the Black Box? Because even when testing a specification, you need to think about what will happen if you enter a value larger than the allowable one, and how the system should react to errors in general. One has to think about this when writing unit tests, which are not related to the Black Box.

About green blocks in the knowledge map

In fact, these names are types of requirements. Tests covering a specific type of requirement are logical to call tests of this requirement. Most of the names of the tests placed in blue blocks also came out of the names of the requirements. For example, “the application must be fault tolerant” generates fault tolerance tests.

Green blocks differ from blue ones in that they are rarely mentioned in Russian-speaking reviews of test types. However, if you, for example, look for the phrase “suitability testing”, you will find many useful things.

How to use the card

My mind map of types of testing, in fact, is a graph, specifically a tree. My favorite XMind application makes it very easy to change the structure of a knowledge map with a tree and other views. But there are many letters in the text, therefore the tree becomes wide and not convenient for perception.

This tree has 10 large branches - the first level of the graph is the classification criteria. Obviously, they are not types of tests themselves.

I hope that you know the difference between a wide search and a deep search. In my deep conviction, errors in understanding and choosing tests occur because of the desire to find a satisfactory solution in a faster way, which pushes to search in depth. 2-level (heading - list) lists of types of testing this greatly contribute.

Need to search in width. Because on each branch of the given graph there are tests that you need. It should also be done when trying to characterize existing test kits. The same test will have several different characteristics.

Spoiler: when you study all types of tests, you will find that in each category there are types of tests you need

Classification criteria

Let's start, let's go over the width of the upper level of the tree.

1. Type of requirements

All tests depend on what is required from the software being developed.

If we have something to develop, then there is something to test. The presence of formally described requirements is a very important thing, but it is not mandatory. It does not matter whether you have a piece of paper called “TK” / “SRS” or not - there are always requirements on the basis of which you are conducting the check.

We can clarify, of course, what operations are needed, or understand and do it according to our vision, but somehow we already have the minimum requirements.

No requirements - no development. The reverse is also true - there is a development -> there are requirements.

We can assume that there are no requirements if the developers did not understand what was expected of them and could not get to work. I know good developers who do not start development until they get an answer to the question: “why should this software do this and that? What is the end use? ”

If the developers work, then there are requirements, another question is the accuracy of their understanding.

For example, “the task is to make a calculator, which will have buttons, and which will perform all mathematical operations” contains a lot of requirements in itself.

Understand your current understanding of what is expected from the software being developed, by the types of requirements that you see in my mind file. If you are aiming for a full test coverage, then

You need all kinds of tests from the first group.

If you need only the "minimum required", and not "necessary and sufficient" - use the functional suitability and accuracy tests.

Details on the functional tests will be discussed in the third part.

At this step, many stop. " We need functional tests ." But we must go further.

2. The object of testing

The most fertile approach to classifying tests is categorization by test object. There is a great variety of technologies, interfaces, architectures, functional purposes and system characteristics. Everything requires its own approach to testing. Different methods and tools.

I will attach this group separately:


On the right, I wrote three groupings, they are conditional , so they are not parent nodes. Because these requirements can be both functional and non-functional, depending on what we develop. In general, we will carry out infrastructure before and for transfer to production. And the operational ones will be carried out on a prod-like environment, when it will be already determined what hardware our system will live on.

Full security testing, for example for certification in the FSB, can be part of infrastructure tests if you use hardware protection methods or external system modules that we do not develop ourselves. And some security tools will be “sewn” into our software and tested as part of the functional requirements.

For example, as part of testing the function of sending a transaction message from an ATM to an issuing bank, we will verify the encryption of the PIN code.

It is very important to understand that

Types of tests for test objects are not types of functional tests.

As well as performance tests, fault tolerance, etc. - not always types of NFR.

Consider the examples.

Example 1. There is a non-functional requirement “if a component fails, its functions must be performed by another, parallel component”. Accordingly, the requirement will be covered by non-functional tests - stress tests, reliability and stability tests.

Example 2. Consider the case where fault tolerance is a functional requirement. For example, a failover cluster is being developed and tested, not a system in which it is used. The purpose of the development is to create a product that will provide fault tolerance. In this case, we deal with functional tests of fault tolerance.

Example 3. Jmeter application development . This is a popular tool for stress testing. Its functionality is load testing. This is a case where the subject of testing has become the object of testing. Recursive, right? JMeter load testing tests are functional.

Other possible examples include the development of a crypto module (functional tests of information security), the development of an interface between systems (functional integration tests), the development of a front-office web interface as a thin client while observing the separation of business logic from data presentation (functional UI tests). And so on.

It is because of such cases that it is not appropriate to divide functional tests by types of functionality (only by types of requirements). And it is not appropriate to classify the functional tests themselves as objects when categorizing “by test object”.

3. Knowledge of the system

Depending on the knowledge of the system, tests are tests of a black, gray and white box. These terms come from Management Theory, and I hope that you are familiar with them in a wider sense than the types of tests.

It is possible to consider both a monochrome box as the entire system as a whole, and its separate part.

The most common approach to testing is to conduct functional black suitability tests . They are also called acceptance tests at the same time, but we have not yet reached acceptance tests, wait. It is on such testing that most novice testers are trained.

It is a common misconception that conducting such tests is necessary and sufficient, and if it does not lead to an increase in the quality of the product, then testers have done a poor job.

My entire series of articles on the selection of tests is aimed at relieving readers of this delusion.

In detail about the impact of knowledge of the system on testing I will write separately.

4. The degree of automation

About automation write everywhere and a lot. I myself love this topic very much.

What I don’t like is the fact that automation is most often considered in this context: “here we have accumulated regression functional tests, there is no one to chase them away, we must somehow have it”. Therefore, many will look at this branch of the tree and say, "well, this later."

Automation is not only an evolutionary development of tests. Some necessary types of tests simply can not be manual.

We need to think about automation when answering the question:

"How do we check this?"

Some situations are simply impossible to reproduce manually. Sometimes we need external system simulators. Sometimes - auxiliary tools for the preparation of test data. Almost always - tools for load testing.
Some tools you take ready, some write yourself. We need to put resources on this, therefore

think about automation in advance.

5. The degree of isolation of components

It is possible to test the “from-and-to” system for safety, for compliance with standards, for convenience of operation. It is about the scale - when you take everything entirely.
You can check the individual parts at different levels of aggregation. You can check not the components themselves, but how they interact - whether data is lost, distorted, or distorted.

The choice of scale on which this or that test will be held depends on the purpose - what mistakes you are looking for, what requirements you want to check. And more on the knowledge and availability of the system.

End-to-end testing can be a black box, for example, on acceptance tests during initial commissioning. And before the transfer - the gray box, when testers know how the system works and where it has bottlenecks. Fill in the data at the front entrance and wait for the flow from all the slots and the expected result from the rear exit itself. From knowledge of the system depends on the preparation of test data. Knowing where it can break through, you can slip something into the right slot. In this case, it may be that a potentially defective component is not accessible on its own - therefore, it is not a question of testing a component. The whole machine is started, and defects can be detected not only where they are expected.

If you isolate a component and only input data to it and check the result only at its output, this will be the component testing.

Depending on how finely the system is crushed, there are discrepancies in terminology regarding the types of testing. This will be discussed in another part.

6. Testing time

When determining the necessary types of tests, you must answer the question

when will we test?

My favorite army anecdote: "Dig from the fence and before sunset." In terms of our work:

"Test from a critical defect to release."

I believe that this approach leads to the fact that it is impossible to achieve an effective test coverage.

The question of the timing of a particular test depends largely on the methodology of the project. According to SCRUM you will take only those tests that you can catch in an iteration. You will have a release date fixed, and you will be able to guess when it is time to make a Smoke, when it makes sense to start a regression, and all the rest of the time you will rake the bug fix and conduct full-fledged tests of new features. Build can be collected every day, and you can drive Smoke and / or regression tests on it. Someone builds a build just before the release and sits at night to verify or double-check everything that can be done.

In my understanding, the trait that separates tests by time is release. Part of the tests is done before the release, inside the team - these are alpha tests, part - after the transfer into operation - these are beta, gamma, delta ... omega tests.

Everyone knows the law of dependence of the cost of a defect on the time of its detection. Therefore, a maximum of tests should be performed on the alpha stage.

Beta tests are usually understood as “prerelease”. When the product seems to be ready, and it is already used, but it is still not complete. The practice of full beta testing is common in the gamedev-industry and in open source-projects.

7. The degree of test readiness

You need to decide:

• You either lay the time to prepare the tests, or - no.
• You either have a common standard for test plans and test reports — or not.
• You are using or are going to use a test management system or not.

Over time, these theses for you may change.

Be that as it may, even if you answered “no” on all items at the beginning, this does not mean that only research testing will be carried out on your project. You can take the TK and check whether what is written in it is done - it will be quite prepared testing. When you know what to look for.

The lack of execution of tests does not mean their unpreparedness.

More - in another part.

8. Depth of testing

I read about Test-to-pass and Test-to-fail in Software Testing By Ron Patton . Here is a quote:

There are two fundamental approaches to the testing software: test-to-pass and test-to-fail. When you test-to-pass, you really only need that software minimally works. You don't push its capabilities. You can not break it. Use the gloves and the most straightforward test cases.

Patton writes that tests that must pass successfully (Test-to-pass) should be checked first. If they did not pass, then the rest can not be checked.

It doesn’t surprise me that this test splitting is hardly mentioned anywhere else. The reason - this characteristic is almost equivalent to the division of test scenarios into positive and negative. In fact, the way it is, except for one moment:

Positive and negative are test cases. With the “test-to-pass” and “test-to-fail” labels you can group test sets for smoke, acceptance and regression tests, which can contain both negative and positive scenarios.

Tets-to-pass are tests in the normal, most frequently used mode of operation.

Test-to-fail are tests in uncharted territory that may be a minefield. You will not carry out these tests after each assembly. They are needed only to search for specific system states in which the occurrence of a previously undetected defect or even a complete system failure is possible. Such tests can be ad hock, and negative tests - these can be cases that are checked when making a bug fix, and they must pass successfully.

The goal of a negative test is to make sure that the system responds correctly to the wrong action. The goal of test-to-fail is to break the system.

Remember that the sooner the defects are found, the better. And that by the end of the iteration, the system should be tested as much as possible, and the basic scenarios should certainly be successful. Thus, we get the alternation of tests in depth.

The first portion of tests should allow detecting a number of show-stopper defects. When the first portion of the Test-To-Pass is successful, go to the second — Test-To-Fail, which should reveal the largest possible number of defects of all degrees of criticality. And after the last portion of tests, defects should not occur at all - they should be Test-To-Pass again.

9. Scripts

About the relevance of the cast in the scheme of the types of scenarios already written above.
Thinking scenarios is not a strategic task. When planning a testing strategy, one thing is certain

scripts need all possible.

Otherwise you will miss the defects.

10. Dynamism

If during testing there are manipulations with the application - it is dynamic. If the state of the system does not change - this is static testing.
Static testing is often overlooked. How can I test without changing anything?
The answer is in the diagram. Code review and documentation testing help identify a significant proportion of errors without wasting time on setting the system in motion.

On this viewing in width is over. Preview in depth leave the next time.

Drawing up a matrix of tests

So, each set of tests can be described by listing 9 characteristics:

• Type of Requirements
• Test object
• Knowledge of the system
• Degree of automation
• The degree of test readiness
• Testing depth
• Testing time
• The degree of isolation of components
• Dynamic

Speaking of a specific project, it will be possible to split all the test objects into functional and non-functional groups, that is, by type of requirements. - 1 measurement.

You will take on one or another set of tests depending on the development stage, so you can group the tests by time. They can be marked as tests that must pass successfully, and as tests by which defects should be detected. - 2 measurements

Conducting static tests can be laid as a standard practice at the organizational level. Code review after each commit with the closure of the ticket, and test the specification when new requirements are received or at the design test stage. - 1 measurement.

Ad hock / research testing should be carried out during outages or in the first days of the project’s life / new functionality. All other tests considered prepared.

There are 4 characteristics of the tests:

• Type of requirements * Test object = Functional component of the system
• Knowledge of the system
• Degree of automation
• The degree of isolation of components

Now it is possible to create a matrix of possible combinations of types of testing.





I here considered that even according to the 4 characteristics of the types of tests it turns out so much that it makes no sense to generate them.

Matrix, matrix ... I consider this very Mind Map with the types of tests to be the matrix of tests. Looking at it you just have to keep in mind the fact that every single test at the same time is a kind of test of all the listed top-level categories.

Conclusion

So many tests are huge. Having an understanding of the purpose of testing and a systematic idea of ​​the types of tests it is already possible to answer the question - what types of tests do you need.

Finally, here's a checklist — questions that need to be answered by determining the right kinds of tests:

1. What functional and non-functional requirements are imposed on the system?
2. What will the system consist of?
3. How well do testers know the structure of the system?
4. How and what to reproduce test situations?
5. In which areas and on what scale will the system be tested?
6. At what stage of development will tests be conducted?
7. How will you describe and store the tests?
8. Do your testers know how to write test scripts?
9. Will the developers check themselves and each other?
10. Is the specification perfect?

(These questions were raised when reviewing test types wide.)

I hope I have not confused readers even more, and my matrix will be useful to you.