Test :: Spec: pros, cons and features

Test :: Spec ( https://metacpan.org/pod/Test::Spec ) is a module for declarative writing of Perl unit tests. We in REG.RU actively use it, so I want to tell why it is needed, how it differs from other modules for testing, to point out its advantages, disadvantages and peculiarities of implementation.

This article is not an introduction to unit testing in general, nor to the use of Test :: Spec in particular. Information on using Test :: Spec can be obtained from the documentation ( https://metacpan.org/pod/Test::Spec and https://metacpan.org/pod/Test::Spec::Mocks ). The article deals with the specifics and nuances of this module.

Table of contents:
Specifications for the code being tested
Unit testing using mock objects
More small utility
Clear exception output
Automatically imports strict / warnings;
Simple and convenient selective test run
There is no alternative
Features work and rake
Displaying test names in ok / is / other does not work
Do not place tests inside before / after
Before / after blocks change code structure
Local operator no longer works
DSL
Global caches
Global variables
How different?
More about it and local
Common code
It is difficult to write helpers that work simultaneously with both Test :: Spec and Test :: More
The with function only works for classes.
The with function does not see the difference between a hash and an array.
Problems testing things like memory leaks
The use_ok function is out of place
Interesting
How objects are technically forged
findings
Test :: Spec is good for unit testing high level code.
')

Specifications for the code being tested

Take a simple test on Test :: More.

Test code:

package MyModule; use strict; use warnings; sub mid { my ($first, $last) = @_; $first + int( ($last - $first) / 2 ); } 1; 

The test itself:

 use strict; use warnings; use Test::More; use MyModule; is MyModule::mid(8, 12), 10, "mid should work"; is MyModule::mid(10, 11), 10, "mid should round the way we want"; done_testing; 

work result:
ok 1 - mid should work
ok 2 - mid should round the way we want
1..2

The equivalent test for Test :: Spec:

 use Test::Spec; use MyModule; describe "MyModule" => sub { describe "mid" => sub { it "should work" => sub { is MyModule::mid(8, 12), 10; }; it "should round the way we want" => sub { is MyModule::mid(10, 11), 10; }; }; }; runtests unless caller; 

and the result of his work:
ok 1 - MyModule mid should work
ok 2 - MyModule mid should round the way we want
1..2

Everything is very similar. Differences in the structure of the test.

Test :: Spec is a way to declare the specifications for the code under test declaratively. This module is created in the likeness of the well-known RSpec package from the Ruby world, which, in turn, works in accordance with the principles of TDD and BDD. The specification for the code being tested describes the functional behavior of the code being tested ( http://en.wikipedia.org/wiki/Behavior-driven_development#Story_versus_specification ). It makes it easier to read the source code of the test and to understand what we are testing and how. At the same time, the lines describing the behavior and the entities with which this behavior corresponds are used when displaying information about successful or failed tests.

Compare these entries:

Ruby:

 describe SomeClass do describe :process do @instance = nil before :all do @instance = SomeClass.new(45) end it "should return to_i" do @instance.to_i.should == 45 end end end 

Perl:

 describe SomeClass => sub { describe process => sub { my $instance; before all => sub { $instance = SomeClass->new(45); }; it "should return to_i" => sub { is $instance->to_i, 45; }; }; }; 

describe - the block where the tests are located (should describe what we are testing). The nesting of the describe blocks is not limited, which allows you to structurally declare the desired behavior in the test and set up test scripts.

it is one separate test (should describe what what we are testing should do). Testing itself takes place inside the “it” blocks, implemented by the usual ok / is / like functions (by default, all functions from Test :: More, Test :: Deep and Test :: Trap are imported).

before / after - allow you to perform various actions before each test, or before each block of tests.

Unit testing using mock objects

Test :: Spec is ideal for unit testing using mock-objects ( https://metacpan.org/pod/Test::Spec::Mocks#Using-mock-objects ). This is its main advantage over other libraries for tests.

To implement unit testing according to the principle “only one module / function is tested at one time”, it is practically necessary to actively use mock objects.

For example, the following method of the User module is an implementation of business logic to provide discounts on purchases:

 sub apply_discount { my ($self, $shopping_cart) = @_; if ($shopping_cart->total_amount >= Discounts::MIN_AMOUNT && Discounts::is_discount_date) { if ($shopping_cart->items_count > 10) { $self->set_discount(DISCOUNT_BIG); } else { $self->set_discount(DISCOUNT_MINI); } } } 

One of the options for testing it could be this: creating a User object ($ self) with all dependencies, creating a basket with the required number of products and with the required amount and testing the result.

In the case of a unit test, only this section of the code is tested, while creating a User and Shopping cart can be avoided.

The test (for one “if” branch) looks like this:

 describe discount => sub { it "should work" => sub { my $user = bless {}, 'User'; my $shopping_cart = mock(); $shopping_cart->expects('total_amount')->returns(4_000)->once; Discounts->expects('is_discount_date')->returns(1)->once; $shopping_cart->expects('items_count')->returns(11); $user->expects('set_discount')->with(Discounts::DISCOUNT_BIG); ok $user->apply_discount($shopping_cart); }; }; 

It uses the functions Test :: Spec :: Mocks: expects , returns , with , once .

The following happens: the User :: apply_discount method is called, the $ shopping_cart mock-object is passed to it. This checks that the total_amount method of the $ shopping_cart object is called exactly once (in fact, no real code will be called — instead, this method will return the number 4000). Similarly, the class method Discounts :: is_discount_date should be called once, and returns one. The items_count method of the $ shopping_cart object will be called at least once and will return 11. And as a result, $ user-> set_discount should be called with the argument Discounts :: DISCOUNT_BIG

That is, in fact, we most naturally check every branch of logic.

This approach gives us the following advantages:

1. The test is easier to write.
2. It is less fragile: if we completely tried to recreate the User object in the test, we would have to deal with breakdowns associated with the fact that the implementation details of something that were not used at all in the tested function changed.
3. The test works faster.
4. Business logic is more clearly stated (documented) in the test.
5. If the bug is in the code, then 100,500 different tests are not falling, but some one, and it is clear from it that what is broken.

If the equivalent unit test had to be written in pure Perl and Test :: More, it would look something like this:

 use strict; use warnings; use Test::More; my $user = bless {}, 'User'; my $shopping_cart = bless {}, 'ShoppingCart'; no warnings 'redefine', 'once'; my $sc_called = 0; local *ShoppingCart::total_amount = sub { $sc_called++; 4_000 }; my $idd_called = 0; local *Discounts::is_discount_date = sub { $idd_called++; 1 }; my $sc2_called = 0; local *ShoppingCart::items_count = sub { $sc2_called++; 11 }; my $sd_called = 0; local *User::set_discount = sub { my ($self, $amount) = @_; is $amount, Discounts::DISCOUNT_BIG; $sd_called = 1; }; ok $user->apply_discount($shopping_cart); is $sc_called, 1; is $idd_called, 1; ok $sc2_called; is $sd_called, 1; done_testing; 

Here it is obvious that a lot of routine work on the falsification of functions is taking place, which could be automated.

More small utility

Clear exception output

 use Test::Spec; describe "mycode" => sub { it "should work" => sub { is 1+1, 2; }; it "should work great" => sub { die "WAT? Unexpected error"; is 2+2, 4; }; }; runtests unless caller; 

gives out:
ok 1 - mycode should work
not ok 2 - mycode should work great
# Failed test 'mycode should work great' by dying:
# WAT? Unexpected error
# at test.t line 8.
1..2
# Looks like you failed 1 test of 2.
which contains, in addition to the line number, the name of the test - “mycode should work great”. Naked Test :: More cannot boast with this and cannot, because the name of the test is not yet known while preparations are coming to it.

Automatically imports strict / warnings;

That is, in fact, it is not necessary to write them. But be careful if you have adopted another code requirements module, for example Modern :: Perl. In this case, turn it on after Test :: Spec.

Simple and convenient selective test run

By simply setting the SPEC = pattern environment variable on the command line, only some tests can be performed. Which is extremely convenient when you debug one test and you do not need the output from the rest.

Example:

 use Test::Spec; describe "mycode" => sub { it "should add" => sub { is 1+1, 2; }; it "should substract" => sub { is 4-2, 2; }; }; runtests unless caller; 

If you run it as SPEC = add perl test.t, then only the “mycode should add” test will be executed.

For more information: https://metacpan.org/pod/Test::Spec#runtests-patterns .

There is no alternative

Modules that allow you to organize test code in a structured way, like RSpec, of course, exist. But there are no alternatives in terms of working with mock-objects.

The creator of the Test :: MockObject module is Chromatic https://metacpan.org/author/CHROMATIC (the author of Modern Perl, participated in the development of Perl 5, Perl 6 and many popular modules on CPAN), does not recognize unit testing, in the documentation for module mock objects are described as "Test :: MockObject - Perl extension for emulating troublesome interfaces" (keyword troublesome interfaces), about which he even wrote a post: http://modernperlbooks.com/mt/2012/04/mock-objects -despoil-your-tests.html

His approach is clearly not for us.

He also noted: “Note: See Test :: MockModule for an alternate (and better) approach”.

Test :: MockModule is extremely inconvenient, not supported (the author is not visible since 2005) and is broken in perl 5.21 ( https://rt.cpan.org/Ticket/Display.html?id=87004 )

Features work and rake

Displaying test names in ok / is / other does not work

More precisely, it works, but it spoils the logic of forming test names in Test :: Spec.

 describe "Our great code" => sub { it "should work" => sub { is 2+2, 4; }; }; 

displays:
ok 1 - Our great code should work
1..1

and code:

 describe "Our great code" => sub { it "should work" => sub { is 2+2, 4, "should add right"; }; }; 

displays:
ok 1 - should add right
1..1

As we see, “Our great code” is lost, which negates the use of text in describe / it.

It turns out, messages in ok and is better not to use.

But what to do if we want two tests in the it block?

 describe "Our great code" => sub { it "should work" => sub { is 2+2, 4; is 10-2, 8; }; }; 

will output:
ok 1 - Our great code should work
ok 2 - Our great code should work
1..2

As you can see, there are no individual messages for each test. If you look carefully at the examples in the Test :: Spec documentation, you can see that each individual test should be in a separate it:

 describe "Our great code" => sub { it "should add right" => sub { is 2+2, 4; }; it "should substract right" => sub { is 10-2, 8; }; }; 

will output:
ok 1 - Our great code should add right
ok 2 - Our great code should substract right
1..2

Which, however, is not very convenient and cumbersome for some cases.

There are problems with other modules made for Test :: More, for example, https://metacpan.org/pod/Test::Exception defaults to an automatically generated message for ok, respectively, you need to explicitly specify an empty string instead.

Do not place tests inside before / after

The before block you will have to use very often, it will initialize the variables before the tests. The block after is basically needed to undo changes made in the outside world, including global variables, etc.

They do not need to try to place the tests themselves, which should be in it. For example:

 use Test::Spec; describe "mycode" => sub { my $s; before each => sub { $s = stub(mycode=>sub{42}); }; after each => sub { is $s->mycode, 42; }; it "should work" => sub { is $s->mycode, 42; }; }; runtests unless caller; 

Gives an error message:
ok 1 - mycode should work
not ok 2 - mycode should work
# Failed test 'mycode should work' by dying:
# Can't locate object method "mycode" via package "Test::Spec::Mocks::MockObject"
# at test.t line 9.
1..2
# Looks like you failed 1 test of 2.

As you can see, in the after block, the mock object created in the before block no longer works. So, if you have a lot of it blocks, and at the end of each block you want to do the same tests, then you won't be able to put them in the after block. You can take them to a separate function, and call it from each of it, but this is similar to the duplication of functionality.

Before / after blocks change code structure

In the example below, we need to initialize a new Counter object for each test (let's imagine that this is difficult and takes many lines of code, so copy / paste is not an option). It will look like this:

 use Test::Spec; use Counter; describe "counter" => sub { my $c; before each => sub { $c = Counter->new(); }; it "should calc average" => sub { $c->add(2); $c->add(4); is $c->avg, 3; }; it "should calc sum" => sub { $c->add(2); $c->add(4); is $c->avg, 3; }; }; runtests unless caller; 

That is, the lexical variable $ c is used, which will be available in the scope of the “it” blocks. Before each of them, the “before” block is called, and the variable is reinitialized.

If you write a similar test without Test :: Spec, you get this:

 use strict; use warnings; use Test::More; use Counter; sub test_case(&) { my ($callback) = @_; my $c = Counter->new(); $callback->($c); } test_case { my ($c) = @_; $c->add(2); $c->add(4); is $c->avg, 3, "should calc average"; }; test_case { my ($c) = @_; $c->add(2); $c->add(4); is $c->sum, 6, "should calc sum"; }; done_testing; 

That is, a callback is passed to the test_case function, then test_case creates a Counter object and calls a callback, passing the created object as a parameter.

In principle, in Test :: More you can organize a test as your heart desires, but the example above is a universal, scalable solution.

If you try to make a tracing with Test :: Spec, a lexical variable that is initialized before each test, you get something “not very correct”:

 use strict; use warnings; use Test::More; use Counter; my $c; sub init { $c = Counter->new(); } init(); $c->add(2); $c->add(4); is $c->avg, 3, "should calc average"; init(); $c->add(2); $c->add(4); is $c->sum, 6, "should calc sum"; done_testing; 

In this code, the function modifies a variable that is not passed to it as an argument, which is already considered a bad style. However, technically it is the same as in the version with Test :: Spec (there, too, the code in the before block modifies a variable that is not explicitly passed to it), but in it it is considered “normal”.

We see that in Test :: More and Test :: Spec the code is organized differently. Different language features are used to organize the test.

Local operator no longer works

More precisely, it works, but not always.

This does not work:

 use Test::Spec; our $_somevar = 11; describe "foo" => sub { local $_somevar = 42; it "should work" => sub { is $_somevar, 42; }; }; runtests unless caller; 

not ok 1 - foo should work
# Failed test 'foo should work'
# at test-local-doesnt-work.t line 8.
# got: '11'
# expected: '42'
1..1
# Looks like you failed 1 test of 1.

So it works:

 use Test::Spec; our $_somevar = 11; describe "foo" => sub { it "should work" => sub { local $_somevar = 42; is $_somevar, 42; }; }; runtests unless caller; 

ok 1 - foo should work
1..1

The thing is that it does not fulfill the callback given to it (or rather, this can already be considered a closure), and remembers the link to it. It is executed during the runtests call. And as we know, local, unlike my, acts "in time" and not "in space".

What problems can this cause? local tests may be needed for two things - to fake a function and fake a variable. Now this is not so easy to do.

In principle, the fact that it is impossible to forge a function with the help of local (and without it it is not practical - you have to return the old function with your hands), only for the benefit. Test :: Spec has its own function falsification mechanism (about it was higher), and there is no other support.

But the impossibility of resetting a variable is worse.

If you are not using local in Perl now, this does not mean that you will not need it in tests. In the next three paragraphs I will tell you why he may be needed.

DSL

The fact is that DSL ( http://www.slideshare.net/mayperl/dsl-perl ) in Perl is very often done using local variables.

For example, we need to retrieve data from the database in the Web application, in the controllers. At the same time, we have configured master / slave replication. By default, the data must be received from the slave servers, but if we are going to modify the received data and write it to the database, the initial data must be received from the master server before the modification.

Thus, we need all of our functions to retrieve data from the database, transfer information: from the slave server to take data or from the master. You can simply transfer them a connection to the database, but this is too cumbersome - there may be many such functions, they can call each other.

Suppose the code for retrieving data from a database looks like this:

 sub get_data { mydatabase->query("select * from ... "); } 

Then we can make the following API: mydatabase will return the connection to the slave database, mydatabase inside the with_mysql_master block will return the connection to the master database.

This is how reading data from a slave looks like:

 $some_data = get_data(); $even_more_data = mydatabase->query("select * from anothertable … "); 

So read data from master and write to master:

 with_mysql_master { $some_data = get_data(); mydatabase->query("insert into … ", $some_data); }; 

The with_mysql_master function is easiest to implement with local:

 our $_current_db = get_slave_db_handle(); sub mydatabase { $_current_db } sub with_mysql_master(&) { my ($cb) = @_; local $_current_db = get_master_db_handle(); $cb->(); } 

Thus, mydatabase inside the with_mysql_master block will return a connection to the master database, because it is in the “action area” of the $ _current_db local override, and outside this block is a connection to the database slave.

So, in Test :: Spec with all such constructions there can be difficulties.

Test :: Spec is made in the image and likeness of Ruby libraries, there DSL is organized without local (and there is no analog of local at all), so this nuance was not foreseen.

Global caches

Look in your state for code. Any use of it can usually be classified as a global cache of something. When they say that global variables are bad, this often applies to such a “non-global” state.

The problem with the state is that it cannot be tested at all (see http://perlmonks.org/?node_id=1072981 ). You cannot call a function from one process many times where something is cached using state and flush caches. We'll have to replace the state with the good old ours. And just when testing to dump it:

 local %SomeModule::SomeData; 

If you need to test such a function with Test :: Spec, and the cache will interfere, you can replace it with two separate functions - the first one returns data without caching (say, get_data), the second one - deals only with caching (cached_get_data). And test only the first one. This will be a unit test (testing one function separately). The second of these functions cannot be tested at all, but this is not really necessary: ​​it is simple - you have to believe that it works.

If you have an integration test that tests a whole call stack, then you will have to forge a cached_get_data call in it and replace it with get_data without caching.

Global variables

Some% SomeModule :: CONFIG is quite normal use-case for using global variables. With local it is convenient to replace the config before calling functions.

If there are difficulties with this in Test :: Spec, it is better to make a function that returns CONFIG and forge it.

How different?

It should be noted that there are modules in which the same structural description of tests is available (even with the same "describe" and "it"), but without this problem with local, for example https://metacpan.org/pod/Test :: Kantan . However, this module, besides the structural description of tests, does not provide any possibilities.

More about it and local

At the beginning of the article, we found out that in each “it” there should be one test. It was originally intended, and the only way it works. What to do if we have a whole cycle, where in each iteration of the test?

It is assumed that the correct way to do this is:

 use Test::Spec; describe "foo" => sub { for my $i (1..7) { my $n = $i + 10; it "should work" => sub { is $n, $i + 10; }; } }; runtests unless caller; 

But since each “it” only remembers the closure, but does not execute it immediately, local cannot be used in this code anymore, such a test fails completely:

 use Test::Spec; our $_somevar = 11; describe "foo" => sub { local $_somevar = 42; for my $i (1..7) { my $n = $i + $_somevar; it "should work" => sub { is $n, $i + $_somevar; }; } }; runtests unless caller; 

not ok 1 - foo should work
# Failed test 'foo should work'
# at t6-local.pl line 9.
# got: '43'
# expected: '12'
not ok 2 - foo should work
# Failed test 'foo should work'
# at t6-local.pl line 9.
# got: '44'
# expected: '13'
not ok 3 - foo should work
# Failed test 'foo should work'
# at t6-local.pl line 9.
# got: '45'
# expected: '14'
not ok 4 - foo should work
# Failed test 'foo should work'
# at t6-local.pl line 9.
# got: '46'
# expected: '15'
not ok 5 - foo should work
# Failed test 'foo should work'
# at t6-local.pl line 9.
# got: '47'
# expected: '16'
not ok 6 - foo should work
# Failed test 'foo should work'
# at t6-local.pl line 9.
# got: '48'
# expected: '17'
not ok 7 - foo should work
# Failed test 'foo should work'
# at t6-local.pl line 9.
# got: '49'
# expected: '18'
1..7
# Looks like you failed 7 tests of 7.

Yes, and everyone “describe” also remembers the closure, but does not fulfill it, so that it is the same as “it”.

Common code

There is a mechanism for connecting common code that can be performed in different tests. Here is the documentation: https://metacpan.org/pod/Test::Spec#spec_helper-FILESPEC , and here is the implementation: https://metacpan.org/source/PHILIP/Test-Spec-0.47/lib/Test/Spec. pm # L354 .

How does he work?

1. It searches for an include file on disk using File :: Spec (bypassing the perl @ INC mechanism and the require file upload mechanism).
2. It loads the file into memory, makes a string with perl code, where the package first changes, then the contents of the read file are “as is” simply included.
3. Executes this line as eval EXPR.
4. The download file itself has the extension .pl, everything works, but it may not be a valid perl file, it may not have enough use, it may not include the wrong paths and so on, respectively, from the point of view of perl there are syntax errors. In general, this is a non-working piece of code that needs to be stored in a separate file.

That is - an absolute hack.

However, it is quite possible to write a common code in the usual way - arrange it as a function, put it into modules:

Test code:

 package User; use strict; use warnings; sub home_page { my ($self) = @_; "http://www.example.com/".$self->login; } sub id { my ($self) = @_; my $id = $self->login; $id =~ s/\-/_/g; $id; } 1; 

Our module with common code for tests:

 package MyTestHelper; use strict; use warnings; use Test::Spec; sub fake_user { my ($login) = @_; my $user = bless {}, 'User'; $user->expects("login")->returns($login); $user; } 1; 

The test itself:

 use Test::Spec; use User; use MyTestHelper; describe user => sub { it "login should work" => sub { my $user = MyTestHelper::fake_user('abc'); is $user->home_page, 'http://www.example.com/abc'; }; it "should work" => sub { my $user = MyTestHelper::fake_user('hello-world'); is $user->id, 'hello_world'; }; }; runtests unless caller; 

The fake_user function creates a User object, at the same time forging the login method of this object, so that it returns the login that we want now (also passed to fake_user). In tests, we check the logic of the User :: home_page and User :: id methods (knowing the login, we know that we need to return these methods). Thus, the fake_user function is an example of reusing code to create a User object and set up fake methods.

It is difficult to write helpers that work simultaneously with both Test :: Spec and Test :: More

As you can see, the test build order for Test :: Spec and Test :: More is very different. Usually, we are unable to write a library that works in both test environments (we do not take into account all sorts of tricks).

For example, we have a helper for Test :: More, which helps in the test to contact Redis.This is necessary for integration testing of code that works with this Redis, as well as conveniently for some other tests (for example, fork tests, where Redis is used to exchange test data between different processes).

This helper gives the following DSL:

 redis_next_test $redis_connection => sub { ... } 

This function executes the code passed as the last argument. A function namespace is available inside the code. Within each redis_next_test block, the namespace is unique. It can and should be used to name Redis keys. At the end of the block, all keys with this prefix are deleted. All this is necessary so that the tests can be executed in parallel with themselves on the CI server, and at the same time they do not spoil each other’s keys, as well as not to litter the developers' cars with unnecessary keys.

A simplified version of this helper:

 package RedisUniqueKeysForTestMore; use strict; use warnings; use Exporter 'import'; our @EXPORT = qw/ namespace redis_next_test /; our $_namespace; sub namespace() { $_namespace }; sub redis_next_test { my ($conn, $cb) = @_; local $_namespace = $$.rand(); $cb->(); my @all_keys = $conn->keys($_namespace."*"); $conn->del(@all_keys) if @all_keys; } 1; 

Sample test with him:

 use strict; use warnings; use Test::More; use RedisUniqueKeysForTestMore; my $conn = connect_to_redis(); # external sub redis_next_test $conn => sub { my $key = namespace(); $conn->set($key, 42); is $conn->get($key), 42; }; done_testing; 

For Test :: Spec, this is no longer suitable, since:

1. The notion of “inside redis_next_test” is completely naturally realized with the help of local, and with local in Test :: Spec problems, as we saw above.
2. Even if redis_next_test didn’t have local, instead of local $ _namespace = $$. Rand () it would be just $ _namespace = $$. Rand () (which would make nested redis_next_test impossible), it wouldn’t work anyway, because $ conn-> del (@all_keys) if @all_keys; would be executed not after the test, but after the callback of the test is added to the internal structures of Test :: Spec (in fact, the same story as with local).

A function that accepts a callback and executes it inside the describe block is suitable, with blocks before (generates a namespace) and after (deletes keys). Here she is:

 package RedisUniqueKeysForTestSpec; use strict; use warnings; use Test::Spec; use Exporter 'import'; our @EXPORT = qw/ describe_redis namespace /; my $_namespace; sub namespace() { $_namespace }; sub describe_redis { my ($conn, $example_group) = @_; describe "in unique namespace" => sub { before each => sub { $_namespace = $$.rand(); }; after each => sub { my @all_keys = $conn->keys($_namespace."*"); $conn->del(@all_keys) if @all_keys; }; $example_group->(); }; } 

And so the test looks like with her:

 use Test::Spec; use RedisUniqueKeysForTestSpec; my $conn = connect_to_redis(); describe "Redis" => sub { describe_redis $conn => sub { it "should work" => sub { my $key = namespace(); $conn->set($key, 42); is $conn->get($key), 42; }; }; }; runtests unless caller; 

The with function only works for classes.

MyModule.pm

 package MyModule; use strict; use warnings; sub f2 { 1 }; sub f1 { f2(42); }; 1; 

Test:

 use Test::Spec; use MyModule; describe "foo" => sub { it "should work with returns" => sub { MyModule->expects("f2")->returns(sub { is shift, 42}); MyModule::f1(); }; it "should work with with" => sub { MyModule->expects("f2")->with(42); MyModule::f1(); }; }; runtests unless caller; 

Result:
ok 1 - foo should work with returns
not ok 2 - foo should work with with
# Failed test 'foo should work with with' by dying:
# Number of arguments don't match expectation
# at /usr/local/share/perl/5.14.2/Test/Spec/Mocks.pm line 434.
1..2
# Looks like you failed 1 test of 2.

Thus, it can only be used to work with class methods, if the perl package is not used as a class, but as a module (procedural programming), it does not work. Test :: Spec simply waits for the first argument $ self, always.

The with function does not see the difference between a hash and an array.

MyClass.pm:

 package MyClass; use strict; use warnings; sub anotherfunc { 1; } sub myfunc { my ($self, %h) = @_; $self->anotherfunc(%h); } 1; 

Test:

 use Test::Spec; use MyClass; describe "foo" => sub { my $o = bless {}, 'MyClass'; it "should work with with" => sub { MyClass->expects("anotherfunc")->with(a => 1, b => 2, c => 3); $o->myfunc(a => 1, b => 2, c => 3); }; }; runtests unless caller; 

Result:
not ok 1 - foo should work with with
# Failed test 'foo should work with with' by dying:
# Expected argument in position 0 to be 'a', but it was 'c'
Expected argument in position 1 to be '1', but it was '3'
Expected argument in position 2 to be 'b', but it was 'a'
Expected argument in position 3 to be '2', but it was '1'
Expected argument in position 4 to be 'c', but it was 'b'
Expected argument in position 5 to be '3', but it was '2'
# at /usr/local/share/perl/5.14.2/Test/Spec/Mocks.pm line 434.
1..1
# Looks like you failed 1 test of 1.

Actually, Perl doesn't see the difference either. And the order of the elements in the hash is undefined. This could be taken into account when developing the API function with and to make a method that facilitates checking hashes.

You can work around this flaw by using returns and checking the data in its callback. For the example above, this would be:

 MyClass->expects("anotherfunc")->returns(sub { shift; cmp_deeply +{@_}, +{a => 1, b => 2, c => 3} }); 

Problems testing things like memory leaks

For example, the stub () function itself is a leak (apparently, stubs are stored somewhere). So this test does not work:

MyModule.pm:

 package MyModule; sub myfunc { my ($data) = @_; ### Memory leak BUG #$data->{x} = $data; ### /Memory leak BUG $data; } 1; 

Test:

 use Test::Spec; use Scalar::Util qw( weaken ); use MyModule; describe "foo" => sub { it "should not leak memory" => sub { my $leakdetector; { my $r = stub( service_id => 1 ); MyModule::myfunc($r); $leakdetector = $r; weaken($leakdetector); } ok ! defined $leakdetector; } }; runtests unless caller; 

This test shows a memory leak, even when it is not.

A test written without a stub works fine (it fills only if the line with the bug in MyModule.pm is uncommented):

 use Test::Spec; use Scalar::Util qw( weaken ); use MyModule; describe "foo" => sub { it "should not leak memory" => sub { my $leakdetector; { my $r = bless { service_id => 1 }, "SomeClass"; MyModule::myfunc($r); $leakdetector = $r; weaken($leakdetector); } ok ! defined $leakdetector; } }; runtests unless caller; 

In any case, since “describe” and “it” remember closures, this in itself can interfere with the search for leaks, since the closure can contain references to all variables that it uses.

The use_ok function is out of place

If you previously used use_ok in tests, now you can say goodbye to it. Judging by the documentation, it can only be used in the BEGIN block (see https://metacpan.org/pod/Test::More#use_ok ), and this is correct, because outside of BEGIN it may not work exactly as in reality ( for example, not to import prototype functions), and it makes no sense to use such a “right” construct for testing imports from modules, violating this import itself.

So, in Test :: Spec it is not customary to write tests outside “it”, and inside “it” the BEGIN block will execute ... as if it were outside of “it”.

So it’s impossible to do everything “beautifully and correctly”, but if you’re not interested in “beautifully and correctly”, then the usual use will do.

Interesting

How objects are technically forged

Separately, it is worth noting how it is technically possible to achieve overlapping of the expects method for any object or class.

This is done by creating a method (surprise!) Expects in the code of the UNIVERSAL package.

Let's try to do the same trick:

 package User; use strict; use warnings; sub somecode {} package main; use strict; use warnings; { no warnings 'once'; *UNIVERSAL::expects = sub { print "Hello there [".join(',', @_)."]\n"; }; } User->expects(42); my $u =bless { x => 123}, 'User'; $u->expects(11); 

will deduce:
Hello there [User,42]
Hello there [User=HASH(0x8a6688),11]
that is, everything works - it was possible to override the method.

findings

Test :: Spec is good for unit testing high level code.

Test :: Spec is good for unit tests, that is, when only one “layer” is tested, and the rest of the functions are faked.

For integration tests, when we are not more interested in fast, convenient and correct testing of a unit of code and all borderline cases in it, but if everything works and everything is correctly “connected”, then Test :: More and analogs are more suitable.

Another criterion is high-level vs low-level code. In high-level code, you often have to test business logic; mock objects are ideal for this. Everything, except the logic itself, is faked, the test becomes simple and clear.

For low-level code, sometimes it makes no sense to write a separate “real” unit test, a separate “integration” one, since in a low-level code there is usually one “layer” and there is nothing to forge. Unit test will be an integration. Test :: More in these cases is preferable because in Test :: Spec there are things that are not very well transferred from the world of Ruby, without taking into account the realities of Perl, and the methods for building code change without good reason.

The unit tests of the high-level code are pretty similar, so the limitations and the listed drawbacks of Test :: Spec are not a big problem for them, and for low-level code and integration tests it is better to leave room for maneuver and use Test :: More.

The article was prepared with the active participation of the development department of REG.RU. Special thanks to SFX , akzhan Ruby dmvaskin Test::Spec, imagostorm , Chips , evostrov , TimurN , nugged , vadiml .