Details and additional features

This article is a translation of the Hypothesis - Details and advanced features page taken from the official manual.

In this part, we consider the less common features of Hypothesis, which you do not need to start using it, but, nevertheless, make your life easier.

Additional test output

Usually the result of a failed test looks like this:

Falsifying example: test_a_thing(x=1, y="foo") 

It will be printed with a repr each named argument.

Sometimes this is not enough, because you have values ​​with repr , which is not very descriptive or because you need to see the result of some intermediate steps of your test. Here is the note function:

 >>> from hypothesis import given, note, strategies as st >>> @given(st.lists(st.integers()), st.randoms()) ... def test_shuffle_is_noop(ls, r): ... ls2 = list(ls) ... r.shuffle(ls2) ... note("Shuffle: %r" % (ls2)) ... assert ls == ls2 ... >>> try: ... test_shuffle_is_noop() ... except AssertionError: ... print('ls != ls2') Falsifying example: test_shuffle_is_noop(ls=[0, 1], r=RandomWithSeed(18)) Shuffle: [1, 0] ls != ls2 

The note is printed in the last test run to include any additional information that may be needed in the test.

Test statistics

If you are using pytest you can see a number of statistics about the tests performed.
passing command line --hypothesis-show-statistics . What will enable
Some general test statistics are:

For example, if you did the following with - --hypothesis-show-statistics :

  from hypothesis import given, strategies as st @given(st.integers()) def test_integers(i): pass 

You will see:

  test_integers: - 100 passing examples, 0 failing examples, 0 invalid examples - Typical runtimes: ~ 1ms - Fraction of time spent in data generation: ~ 12% - Stopped because settings.max_examples=100 

The final line "Stopped because" is especially important for note: it tells you the setting value that determines when the test should stop trying new examples. This can be useful for understanding the behavior of tests. Ideally, you always want to have max_examples .

In some cases (for example, in filters and recursive strategies) you will see events that describe some aspects of data generation:

  from hypothesis import given, strategies as st @given(st.integers().filter(lambda x: x % 2 == 0)) def test_even_integers(i): pass 

In the end, get something like:

  test_even_integers: - 100 passing examples, 0 failing examples, 36 invalid examples - Typical runtimes: 0-1 ms - Fraction of time spent in data generation: ~ 16% - Stopped because settings.max_examples=100 - Events: * 80.88%, Retried draw from integers().filter(lambda x: <unknown>) to satisfy filter * 26.47%, Aborted test because unable to satisfy integers().filter(lambda x: <unknown>) 

You can also mark custom events in the test using the event function:

hypothesis.event (value)

  from hypothesis import given, event, strategies as st @given(st.integers().filter(lambda x: x % 2 == 0)) def test_even_integers(i): event("i mod 3 = %d" % (i % 3,)) 

Then you will see the result:

  test_even_integers: - 100 passing examples, 0 failing examples, 38 invalid examples - Typical runtimes: 0-1 ms - Fraction of time spent in data generation: ~ 16% - Stopped because settings.max_examples=100 - Events: * 80.43%, Retried draw from integers().filter(lambda x: <unknown>) to satisfy filter * 31.88%, i mod 3 = 0 * 27.54%, Aborted test because unable to satisfy integers().filter(lambda x: <unknown>) * 21.74%, i mod 3 = 1 * 18.84%, i mod 3 = 2 

event arguments can be of any type, but two events will be considered the same if they are the same when converted to a string with str .

Using assumptions

Sometimes Hypothesis does not give you exactly the right kind of data you want - it’s basically the right shape. This is not particularly scary, but some examples will fail, and you do not want to take care of them. You can simply ignore these cases, interrupting the test earlier, but at the same time there is a risk of accidentally missing important things and experiencing much less than you expected. It would also be nice to spend less time on bad examples - if you use 100 examples per test (by default), and it turns out that 70 of these examples do not match your needs, it turns out that a lot of time is wasted.

hypothesis.assume (condition)

For example, suppose you had the following code:

  @given(floats()) def test_negation_is_self_inverse(x): assert x == -(-x) 

Doing it will give us:

  Falsifying example: test_negation_is_self_inverse(x=float('nan')) AssertionError 

It is annoying. We (perhaps) know something about NaN, but in this episode we would not like to recall it and how to handle this situation, but as soon as Hypothesis finds an example of NaN, he will drop everything and hurry to tell us about it. The test will fail and spoil us with all the statistics, and we want to pass it.

So let's block this particular example:

 from math import isnan @given(floats()) def test_negation_is_self_inverse_for_non_nan(x): assume(not isnan(x)) assert x == -(-x) 

This version of the code already passes without problems.

In order to exclude an easy supposed interception, you can assume much more than you intended and the Hypothesis will fail the test if it cannot find enough examples undergoing assumption.

If we wrote:

 @given(floats()) def test_negation_is_self_inverse_for_non_nan(x): assume(False) assert x == -(-x) 

Then at start we would have an exception:

 Unsatisfiable: Unable to satisfy assumptions of hypothesis test_negation_is_self_inverse_for_non_nan. Only 0 examples considered satisfied assumptions (*  (assumptions)  hypothesis test_negation_is_self_inverse_for_non_nan.  0     (assumptions)*) 

How correctly to assume?

Hypothesis has an adaptive intelligence strategy to try to avoid cases that falsify assumptions and, as a rule, lead to the fact that you can still find examples in hard-to-reach situations.

Suppose we had the following:

 @given(lists(integers())) def test_sum_is_positive(xs): assert sum(xs) > 0 

It is not surprising that such a test will fail and produce a falsifying example [] .

Adding assume(xs) to this will remove the trivial empty example and give us [0] .

Add assume(all(x > 0 for x in xs)) and, oh, a miracle! he passes! Indeed, the sum of positive is greater than zero!

Surprising is not that he does not find a counter example, but that he finds enough examples at all.

To make sure something interesting happens, try it on long lists. For example, add assume(len(xs) > 10) . In principle, this should never be an example: a primitive strategy will find less than one in a thousand examples, because if each element of the list is negative with half probability, you will have to get ten of them, by chance. In the default configuration, the hypothesis surrenders long before she tried 1000 examples (by default, she tries 200).

Here is what happens if we try to run it:

 @given(lists(integers())) def test_sum_is_positive(xs): assume(len(xs) > 10) assume(all(x > 0 for x in xs)) print(xs) assert sum(xs) > 0 In: test_sum_is_positive() [17, 12, 7, 13, 11, 3, 6, 9, 8, 11, 47, 27, 1, 31, 1] [6, 2, 29, 30, 25, 34, 19, 15, 50, 16, 10, 3, 16] [25, 17, 9, 19, 15, 2, 2, 4, 22, 10, 10, 27, 3, 1, 14, 17, 13, 8, 16, 9, 2... [17, 65, 78, 1, 8, 29, 2, 79, 28, 18, 39] [13, 26, 8, 3, 4, 76, 6, 14, 20, 27, 21, 32, 14, 42, 9, 24, 33, 9, 5, 15, ... [2, 1, 2, 2, 3, 10, 12, 11, 21, 11, 1, 16] 

As you can see, Hypothesis does not find many examples, but some are quite sufficient to get a successful result.

In general, if you can allow yourself to more accurately form your strategies for your tests, then you should use it - for example, integers(1, 1000) much better than assume(1 <= x <= 1000) .

Strategy definition

The type of object that is used to study the examples provided by your test function is called hypothesis.SearchStrategy .
They are created using functions opened in the hypothesis.strategies module.

Many of these strategies provide various arguments that can be used to customize the generation. For example, for integers, you can specify the min and max values ​​of the integers that you require. If you want to see what exactly the strategy will produce, you can request an example:

 >>> integers(min_value=0, max_value=10).example() 1 

Many strategies are built from other strategies. For example, if you want to define a tuple, you need to say what happens in each element:

 >>> from hypothesis.strategies import tuples >>> tuples(integers(), integers()).example() (-24597, 12566) 

Additional information: doc: available in a separate document <data> .

In-depth information about the set parameters

hypothesis.given (* given_arguments, ** given_kwargs)

The @given decorator can be used to specify which function arguments should be parameterized. You can use positional or named type arguments or their mix.

For example, all of the following are valid:

 @given(integers(), integers()) def a(x, y): pass @given(integers()) def b(x, y): pass @given(y=integers()) def c(x, y): pass @given(x=integers()) def d(x, y): pass @given(x=integers(), y=integers()) def e(x, **kwargs): pass @given(x=integers(), y=integers()) def f(x, *args, **kwargs): pass class SomeTest(TestCase): @given(integers()) def test_a_thing(self, x): pass 

The following are not:

  @given(integers(), integers(), integers()) def g(x, y): pass @given(integers()) def h(x, *args): pass @given(integers(), x=integers()) def i(x, y): pass @given() def j(x, y): pass 

The rules for determining what is valid using given are as follows:

1. You can pass any named argument given .
2. Positional arguments given equivalent to the rightmost named arguments for the test function.
3. Positional arguments cannot be used if the base test function has variable-length arguments (varargs), arbitrary keywords or arguments intended only for keywords.
4. Functions tested with given may have no default values.

The reason for the behavior of the " @given named arguments" is that @given using instance methods: self will be passed to the function as normal and will not be parameterized.

The function returned by given has all the same arguments as the original test, minus those filled with @given .

Execution of custom functions

Hypothesis provides you with a tool that allows you to control how it runs examples.

This allows you to perform actions such as setting up and disassembling each example, executing examples in a subprocess, converting coroutine tests into regular tests, etc. For example, TransactionTestCase in
Django extra runs each example in a separate database transaction.

Thus, introducing the concept of executor or in Russian artist . executor is, in essence, a function that takes a block of code and starts it. The default executor is:

 def default_executor(function): return function() 

You define the performers by defining the execute_example method in the class. Any test methods used in this class with the @given decorator will use self.execute_example as the test performer. For example, the following executor runs all of its code twice:

 from unittest import TestCase class TestTryReallyHard(TestCase): @given(integers()) def test_something(self, i): perform_some_unreliable_operation(i) def execute_example(self, f): f() return f() 

Note: the functions that you use in the map, etc. Will work inside the artist. those. they will not be called until the function passed to execute_example called.

The executor must be able to process the passed function, which returns None, otherwise he will not be able to run the usual test cases. For example, the following artist is not valid:

 from unittest import TestCase class TestRunTwice(TestCase): def execute_example(self, f): return f()() 

and should be rewritten as:

 from unittest import TestCase class TestRunTwice(TestCase): def execute_example(self, f): result = f() if callable(result): result = result() return result 

Using Hypothesis to Search for Values

You can use the Hypothesis data mining functions to find values ​​that satisfy a certain predicate (selection condition). This is usually useful for exploring custom strategies defined using: @composite , or experimenting with data filtering conditions.

hypothesis.find (specifier, condition, settings = None, random = None, database_key = None)

 >>> from hypothesis import find >>> from hypothesis.strategies import sets, lists, integers >>> find(lists(integers()), lambda x: sum(x) >= 10) [10] >>> find(lists(integers()), lambda x: sum(x) >= 10 and len(x) >= 3) [0, 0, 10] >>> find(sets(integers()), lambda x: sum(x) >= 10 and len(x) >= 3) {0, 1, 9} 

The first argument hypothesis.find describes the data in the usual way for the hypothesis.given argument, and supports all the same data types <data> . The second is the predicate that it must satisfy.

Of course, not all conditions are met. If you request an example from Hypothesis with a condition that is always false, it will cause an error:

 >>> find(integers(), lambda x: False) Traceback (most recent call last): ... hypothesis.errors.NoSuchExample: No examples of condition lambda x: <unknown> 

( lambda x: unknown is due to the fact that Hypothesis cannot get the source code of the lambda expression from the python interactive console.)

Inferred Strategies

In some cases, the hypothesis may decide what to do when you omit the arguments. It is based on introspection, not on magic, and therefore has clearly defined limits.

hypothesis.strategies.builds() check the target signature (In python3.6 inspect.getfullargspec() ). If there are mandatory arguments with type annotations and the strategy was not passed to hypothesis.strategies.builds() , then hypothesis.strategies.from_type() used to fill them. You can also pass the special value hypothesis.infer() as an argument to push the arguments with the default value to this output.

 >>> def func(a: int, b: str): ... return [a, b] >>> builds(func).example() [-6993, ''] 

@given does not perform any implicit output for the required arguments, as this would violate compatibility with the pytest functionality.
hypothesis.infer can be used as a keyword argument to explicitly fill in an argument from a type annotation.

 @given(a=infer) def test(a: int): pass # is equivalent to @given(a=integers()) def test(a): pass 

Restrictions

PEP 3107 annotations are not supported in Python 2, and Hypothesis does not check PEP 484 comments at run time.

While hypothesis.strategies.from_type will work as usual, the output in
hypothesis.strategies.builds and @given will work only if you manually create the attribute __annotations__ (for example, using the decorators @annotations(...) and @returns(...) ).

The typing module is fully supported in Python 2, if you have a backport installed.

The typing module is temporary and has a number of internal changes between Python 3.5.0 and 3.6.1, including minor versions. All of them are supported, but there may be problems with the old version of the module. Please let us know about them and consider upgrading to a newer version of Python as a workaround.

Back Next