C ++ tests without macros and dynamic memory

Many popular libraries for testing, such as Google Test, Catch2, Boost.Test, are heavily tied to using macros, so as an example of tests on these libraries, you usually see a picture like this:

namespace { // Tests the default c'tor. TEST(MyString, DefaultConstructor) { const MyString s; EXPECT_STREQ(nullptr, s.c_string()); EXPECT_EQ(0u, s.Length()); } const char kHelloString[] = "Hello, world!"; // Tests the c'tor that accepts a C string. TEST(MyString, ConstructorFromCString) { const MyString s(kHelloString); EXPECT_EQ(0, strcmp(s.c_string(), kHelloString)); EXPECT_EQ(sizeof(kHelloString)/sizeof(kHelloString[0]) - 1, s.Length()); } // Tests the copy c'tor. TEST(MyString, CopyConstructor) { const MyString s1(kHelloString); const MyString s2 = s1; EXPECT_EQ(0, strcmp(s2.c_string(), kHelloString)); } } // namespace 

To macros in C ++, the attitude is wary, why are they so flourishing in libraries for creating tests?

The unit test library should provide its users with a way to write tests, so that the test runtime environment can somehow find and execute them. When you think about how to do this, then using macros seems the easiest. The TEST () macro usually defines a function somehow (in the case of the Google Test, the macro also creates a class) and ensures that the address of this function falls into some global container.

A well-known library in which the approach is implemented without a single macro is the tut-framework . Let's see her example from the tutorial:

 #include <tut/tut.hpp> namespace tut { struct basic{}; typedef test_group<basic> factory; typedef factory::object object; } namespace { tut::factory tf("basic test"); } namespace tut { template<> template<> void object::test<1>() { ensure_equals("2+2=?", 2+2, 4); } } 

The idea that lies at the core is quite interesting and it works, it’s all not very complicated. In short, you have a base class in which the template function is implemented, which assumes parameterization with an integer number:

 template <class Data> class test_object : public Data { /** * Default do-nothing test. */ template <int n> void test() { called_method_was_a_dummy_test_ = true; } } 

Now when you write this test:

 template<> template<> void object::test<1>() { ensure_equals("2+2=?", 2+2, 4); } 

You actually create a test method specialization for a specific number N = 1 (this is exactly what the template<>template<> ). By calling test<N>() the test execution environment can understand whether it was a real test or it was a stub looking at the value called_method_was_a_dummy_test_ after the test was executed.

Further, when you declare a group of tests:

 tut::factory tf("basic test"); 

First, you enumerate all test<N> to a certain constant wired into the library, and, secondly, as a side effect, add information about the group to the global container (group name and addresses of all test functions).

Tut uses exceptions as test conditions, so the tut::ensure_equals() function will simply throw an exception if the two values ​​passed to it are not equal, and the test run environment will catch the exception and count the test as failed. I like this approach; any C ++ developer will immediately understand where such asserts can be used. For example, if my test created an auxiliary stream, then it is useless to place asserts there, no one will catch them. In addition, I understand that my test should be able to free up resources in the event of an exception, as if it were a normal exception-safe code.

In principle, the tut-framework library looks pretty good, but there are some drawbacks to its implementation. For example, for my case, I would like the test to have not only a number, but also other attributes, in particular the name, as well as the "size" of the test (for example, is it an integration test or is it a unit test). This can be solved within the tut API, and even something is already there, but something can be realized by adding a method to the library API, and calling it in the body of the test to set some of its parameters:

 template<> template<> void object::test<1>() { set_name("2+2"); // Set test name to be shown in test report ensure_equals("2+2=?", 2+2, 4); } 

Another problem is that the tut test runner knows nothing about such an event as the beginning of a test. The environment performs object::test<N>() and it does not know in advance whether the test is implemented for this N, or is it just a stub. She only learns when the test is over by analyzing the value of the called_method_was_a_dummy_test_ . This feature does not show itself very well in CI systems that can group the output that the program made between the beginning and the end of the test.

However, in my opinion the main thing that can be improved ("fatal flaw") is the presence of unnecessary auxiliary code required for writing tests. In the tutorial tut-framework, there are quite a lot of things: it is proposed to first create a certain class struct basic{} , and describe tests as object methods related to this. In this class, you can define the methods and data that you want to use in a test group, and the constructor and destructor frame the test execution, creating such a thing as fixture from jUnit. In my practice of working with tut, this object is almost always empty, but it drags a certain number of lines of code.

So, we go to the bicycle workshop and try to shape the idea in the form of a small library.

Here is the minimum test file in the "tested" library:

 // Test group for std::vector (illustrative purposes) #include "tested.h" #include <vector> template<> void tested::Case<CASE_COUNTER>(tested::IRuntime* runtime) { runtime->StartCase("emptiness"); std::vector<int> vec; tested::Is(vec.empty(), "Vector must be empty by default"); } template<> void tested::Case<CASE_COUNTER>(tested::IRuntime* runtime) { runtime->StartCase("AddElement"); std::vector<int> vec; vec.push_back(1); tested::Is(vec.size() == 1); tested::Is(vec[0] == 1); tested::FailIf(vec.empty()); } void LinkVectorTests() { static tested::Group<CASE_COUNTER> x("std.vector", __FILE__); } 

In addition to the absence of macros, the bonus is the absence of the use of dynamic memory inside the library.

Test Case Definition

For registration of tests the template magic of initial level is used on the same principle as tut. Somewhere in tested.h there is a template function of this type:

 template <int N> static void Case(IRuntime* runtime) { throw TheCaseIsAStub(); } 

Test cases that library users write are simply specializations of this method. The function is declared static, i.e. in each translation unit we create specializations that do not intersect each other by name when linking.

There is such a rule that you first need to call StartCase() , which you can pass things like the name of the test and maybe some other things that are still in development.

When the test calls runtime->StartTest() interesting things can happen. First of all, if the tests are now in run mode, then you can report somewhere that the test has started execution. Secondly, if now there is a mode of collecting information about the available tests, StartTest() throw a special kind of exception that will mean that the test is real, and not a stub.

check in

At some point, you need to collect the addresses of all test cases and somewhere to put them. In tested, this is done using groups. This is done by the constructor of the class tested :: Group as a side effect:

 static tested::Group<CASE_COUNTER> x("std.vector", __FILE__); 

The constructor creates a group with the specified name and adds to it all Case<N> cases that it finds in the current translation unit. It turns out that in one translation unit you cannot have two groups. It also means that you cannot split one group into multiple translation units.

The template parameter goes to how many test cases to look for in the current translation unit for the group being created.

Linking

In the above example, the creation of the object tested :: Group () occurs inside a function that we have to call from our application to register the tests:

 void LinkStdVectorTests() { static tested::Group<CASE_COUNTER> x("std.vector", __FILE__); } 

The function is not always required, sometimes you can simply declare an object of the class tested::Group inside the file. However, my experience is that the linker sometimes "optimizes" the entire file, if it is compiled inside the library, and none of the main application uses any characters from this cpp file:

 calc.lib <- calc_test.lib(calc_test.cpp) ^ ^ | | app.exe run_test.exe 

When the calc_test.cpp is not linked to the run_test.exe sources, the linker simply removes this file from consideration in its entirety, together with the creation of a static object, despite the fact that it has the necessary side effects.

If what chain leads from run_test.exe, then a static object will appear in the executable file. And no matter how it will be done, as in the example:

 void LinkStdVectorTests() { static tested::Group<CASE_COUNTER> x("std.vector", __FILE__); } 

or so:

 static tested::Group<CASE_COUNTER> x("std.vector", __FILE__); void LinkStdVectorTests() { } 

The first option, in my opinion, is better because the constructor call is executed after the start of the main () operation, and the application has some control over this process.

I think that this installation of crutches is required for any unit-testing library that uses global variables and constructor side effects to create a database of tests. However, it can probably be avoided by linking the test library with the key --whole-archive (the analogue in MSVC appeared only in Visual Studio 2015.3).

Macros

I promised that there will be no macros here, but it is - CASE_COUNTER . The working version is that it is used by __COUNTER__ , a macro that the compiler increases by one each time it is used inside the translation unit.
GCC, CLANG, MSVC is supported, but not standard. If this is frustrating, here are some alternatives:

• use digits 0, 1, 2
• use standard __LINE__ .
• use constexpr magic level 80. You can search for "constexpr counter" and try to find a compiler on which it will work.

The problem with __LINE__ is that using large numbers in the template parameters creates a large executable file size. That is why I limited the signed char template type to 128 as the maximum number of tests in a group.

Waiver of dynamic memory

It turned out that when registering tests, you can not use dynamic memory, which I used. It is possible that there is no dynamic memory in your environment or you are searching for memory leaks in test cases, so that the intervention of the test execution environment is not what you need. Google Test is struggling with this, here is a fragment from there:

 // Use the RAII idiom to flag mem allocs that are intentionally never // deallocated. The motivation is to silence the false positive mem leaks // that are reported by the debug version of MS's CRT which can only detect // if an alloc is missing a matching deallocation. // Example: // MemoryIsNotDeallocated memory_is_not_deallocated; // critical_section_ = new CRITICAL_SECTION; class MemoryIsNotDeallocated 

And we can simply not create difficulties.

How do we then get a list of tests? These are more technical insides that are easier to see in the source code, but I’ll tell you anyway.

When a group is created, its class will receive a pointer to the function tested::CaseCollector<CASE_COUNTER>::collect , which will collect all the tests of the translation unit into the list. This is how it works:

 // Make the anonymouse namespace to have instances be hidden to specific translation unit namespace { template <Ordinal_t N> struct CaseCollector { // Test runtime that collects the test case struct CollectorRuntime final : IRuntime { void StartCase(const char* caseName, const char* description = nullptr) final { // the trick is exit from test case function into the collector via throw throw CaseIsReal(); } }; // Finds the Case<N> function in current translation unit and adds into the static list. It uses the // reverse order, so the case executed in order of appearance in C++ file. static CaseListEntry* collect(CaseListEntry* tail) { CaseListEntry* current = nullptr; CollectorRuntime collector; try { Case<N>(&collector); } catch (CaseIsStub) { current = tail; } catch (CaseIsReal) { s_caseListEntry.CaseProc = Case<N>; s_caseListEntry.Next = tail; s_caseListEntry.Ordinal = N; current = &s_caseListEntry; } return CaseCollector<N - 1>::collect(current); } private: static CaseListEntry s_caseListEntry; }; // This static storage will be instantiated in any cpp file template <Ordinal_t N> CaseListEntry CaseCollector<N>::s_caseListEntry; } 

It turns out that in each translation unit there are many static variables of the form CaseListEntry CaseCollector \ :: s_caseListEntry, which are elements of the test list, and the collect () method collects these elements into a single-linked list. Approximately in the same way the list forms test groups, but without patterns and recursion.

Structure

Tests need a different binding, such as output in the console with red letters Failed, creating test reports in a format understandable for CI or GUI in which you can view a list of tests and run selected ones - in general, a lot of things. I have a vision of how this can be done, which is different from what I saw before in the testing library. The claim is mainly to libraries that call themselves "header-only", while including a large amount of code, which in fact is not at all for header files.

The approach that I assume is that we divide the library into a front-end — this one itself is tested.h and the back-end are libraries. For writing tests, only tested.h is needed, which is now C ++ 17 (because of the awesome std :: string_view) but it is assumed that it will be C ++ 98. Tested.h performs the actual registration and search for tests, the minimally convenient launch option, and the ability to export tests (groups, addresses of test case functions). Back-end libraries that do not exist yet can do everything they need, in terms of output and launch, using the export functionality. In the same way, you can adjust the launch to the needs of your project.

Total

The library tested ( github code ) still has some stabilization. In the near future plans to add the ability to perform asynchronous tests (needed for integration tests in WebAssembly) and specify the size of the tests. In my opinion, the library is still not quite ready for production use, but I have already unexpectedly spent a lot of time and this stage has come to stop, take a breath and ask for feedback from the community. Would you be interested in using this kind of library? Maybe in C ++ arsenal there are any other ideas on how to create a library without macros? Is such a problem statement interesting at all?