Accident in auto tests

Introduction

When I wrote my first autotest a few years ago, it looked like this. In a cycle, I took out a random user from the database 100 times, performed the required operation on it, and checked that the result was fine for me. It seemed logical enough: I can not test on a single user, this is not enough, it will not prove anything.

A considerable time has passed since then, I managed to work on several different projects in different languages ​​and even change the team. Today I can say with confidence: you should not use chance in your autotests , except in cases that will be specified separately. And I will tell you why.

Example

For examples I will use the simplest function that squares a number but retains the sign. On Ruby, for example, it would look like this:

def smart_sqr(x) x > 0 ? x*x : -x*x; end 

It is easy to imagine what the test will look like for such a function. I'll just take some test cases and compare the value of smart_sqr() on these examples with test cases:
')

 assert_equal(smart_sqr(4), 16); 

The question is - on what basis should I choose the values?

"Benefits" of random values

Why I began to choose random values ​​at the time when I wrote my first autotest? Why do programmers continue to use random values ​​in their tests? Their (and my) logic is easy to understand: one experiment does not prove anything, the approach is purely probabilistic: the more different options tested, the better.

It's not quite like that. As a rule, in modern systems the theoretical proof of the fidelity of the programs (a) is almost impossible and (b) is not required. The entire program is based on the hypothesis of the programmer himself that she is doing what she should. It is impossible to prove this hypothesis, but with the help of tests I can reduce my program to a set of simpler hypotheses, an informal understanding of which would be more accessible.

What I mean? For the function written above, it is obvious to me, in a sense, that it behaves the same on all positive numbers. By “obviously,” I mean the very hypothesis on which my belief is built that my program works as it should (this is a common problem in all engineering disciplines, that some things have to be done). In the absence of any hypotheses, any testing would be useless; only formal proof (which, I repeat, on the verge of the impossible) would help me.

In the presence of a hypothesis, it suffices to test the function of the function on only one positive number in order to make sure that it functions correctly at all.

So, I do not need random values ​​to test the performance of my function. I simply use all the boundary values ​​(or rather, those that seem to me to be such) and one value for each class of values, which, in my opinion, behave the same. In reality, for our function, I would use the values ​​–7, 0 and 13. Your opinion on the boundary conditions may differ from mine, and this is normal. For example, a unit behaves in a slightly different way: its square is equal to the initial value.

Also, many programmers may feel that it is pointless to run the test on all the same values ​​again and again, because their results can not change. This is true, but the task of autotests is not to look for errors in already running programs, their task is to respond to code changes. If you do not change the code, then you can not re-run the tests at all.

Disadvantages of random values

If you use random values ​​in tests, you may encounter a number of problems.

First, the test may behave inconsistently. This is theoretically unacceptable and can also cause a lot of problems in practice (for example, your system running tests may decide that you broke and fixed everything just because the test blinked red and perform some unwanted actions). The test should respond to code changes and only to him. The fall of the tests due to the disturbance of the environment is already a problem, there is no need to aggravate it, increasing the influence of the environment by adding tests that depend on the state of the random number generator.

Secondly, debugging such tests can be a serious problem. If the values ​​on which the test fell were not preserved, then such a result may turn out to be useless.

Thirdly, the test code may lose its clarity when random numbers are added to it. What should be the square of a random number in our example? The square of this random number? With this approach, the test code will exactly repeat the function code (oh, by the way, a great idea, we will use it for verification!).

But in my case ...

Yes, in some cases, the use of random values ​​may be useful. But you must be extremely wary of this. In some languages, it is possible to invoke class private methods. And this, too , can sometimes be useful. But this is not a reason not to think seven times, and then two more before using this opportunity.

I will give a couple of cases in which, in my opinion, you can close your eyes to the use of random values. This is not a complete list. If common sense tells you that you can or should even break the rule that I voiced above, break it.

You are looking for a mistake. You know that there is an error in your code, it sometimes manifests itself in production. Detect it, following the logic, can not. You can try to find it by searching. To do this, you can use your automatic testing system, which will check a certain number of random values ​​each time it starts. In this solution, all is well, except that it is not quite an autotest. This is just an error search script that you have integrated into your automated testing system for convenience. Think twice: you may not need this integration at all.

Source data is too large. It may happen that you are relatively indifferent to the source data, but their volume is such that storing them is difficult. In this case, you can create them on the fly, although storing pregenerated data is still preferable. There is also an option with automatic non-random generation.

If you still decide to use random values ​​in your tests, you need to save the value with which the random number generator was initialized. If you use randomness in third-party modules or systems (for example, inside your database), this can cause serious technical difficulties.

Finally

I would like to add that automatic testing, in my opinion, is one of the most poorly studied and formalized areas in programming. Diametrically opposite answers can be obtained for any question, and for any reason you can hear conflicting opinions. Even the points of view of respected and recognized professionals can vary considerably. If you try to find the answer to the question discussed in my article right now, you will hear thousands of points of view, ranging from “randomness is necessary” and ending with “randomness is unacceptable.” I tried to explain my ideas as clearly as possible, since the simple formulation of principles in the field of auto-testing has long ceased to work.