Specification Testing
Recently, I wrote about various types of testing and the fact that integration testing is convenient to produce using specifications. In this article I will show exactly how this testing happens.
The specification is a text file describing what to test in the test data. It indicates what results the program should receive. The test code finds the real, calculated on the live code results. And the test engine performs verification of the specification and the calculated results.
This approach allows you to declaratively create tests. Specifications are easy to read and supplement when requirements change. The test code is compact. It is easy to maintain and expand.
')
The article describes the principles of the engine for testing specifications and provides examples of use. The engine itself is attached to the article. It can be considered a small library for integration testing.
Source code - stalker98.narod.ru/TestBySpecification.rar
The code is commented in detail, so if after reading the article some points remain unclear, the details can be viewed in the implementation. By and large, the library consists of only 4 classes, which is not much. The testing engine is in Utils.Common.Tests.
For testing, I use the 2008 studio test framework, but can be transferred to NUnit. To do this, you need to correct the test call in Test.Domain.Documents.DocumentTester.cs and Test.Domain.Polygons.PolygonTester.cs.
Let the task is to write a document converter from one format to another. The conversion is tricky, with a lot of mathematical calculations. The most difficult part of the transformation is the processing of geometric shapes described in the documents. The customer handed over a set of standard documents that he needs to translate into another format.
The development is still far away. Already written code is covered by unit-tests, each of which tests its module in isolation from the rest of the program. But there comes a time when it is required to make sure that the written code will work on real data. Classes of the program in this case should be used together, without isolation from each other. It is necessary to conduct integration testing.
The testing approach is the same - for each sample document sent by the customer, we will create a specification, where we write down the significant results that our program will receive when converting. From the sent documents we will choose various figures and also we will write for them the specifications. In the specifications of the figures we will write down the results that our program should come to when processing a particular figure.
The specification is a text file that describes what to test in the test data. The file consists of:
To check the compliance of properties from the specification and real (calculated) property values, you need to know how to find these real property values. For this, a heir of the Specification class is created and get accessors for properties are written in it, with the same names as in the specification:
Property type must match the name. For example, properties that begin with Is, Has, or Are are considered by the library as Boolean flags. And properties ending in Count are integers. The mechanism of matching the name of a property to its type will be described below.
As you can see, the test code is minimized.
Launch testing in the studio Ctrl + R, A:
When testing, a log of operations is conducted (it is also a test output). What specifications and properties have been written to the log, what the test result is, how long the testing took. If a property has not been tested, this is indicated separately. Test output can be viewed by clicking on the test in the Test Results window.
I wrote 5 more specifications for various shapes. When testing, it turned out that the IsRhombus property (a boolean flag - whether the figure is a diamond) in the specifications does not match the value that is in the program. Obviously, an error was made in the method that determines whether the figure is a diamond. The log looks like this:
For an example, I took some simple checks, so everywhere in the computation time it is 00: 00: 00.000. In real projects, time will be more significant.
The attentive reader at the moment should not understand two questions - where do the test data come from and what to do if you need to test something other than the equality of two bool or int. On both questions the answer lies in the mechanism for determining properties.
Each property belongs to one specific type. Moreover, the type here means more than just the .NET type. It is rather .NET type plus behavior when testing. The behavior is described by a special object - the property descriptor PropertyDescriptor <T>. The descriptor defines the convention adopted for property names, the conversion from string to property value and vice versa, as well as the criterion that determines whether the property passed the test or not.
I will cite as an example the library descriptor for Boolean flags:
The descriptor specifies the regular expression NamePattern , which specifies a convention for naming properties to which the descriptor applies. In this case, these are all properties with names that begin with Is, Has, or Are. Convert sets the function to convert from string to property value. Verify defines the criteria for passing the test. In this case, it is a simple test for equality. That is, if a Boolean value in the specification is equal to the calculated value, then it is considered that the property passed the test. If Verify is omitted, the property will be read, but will not be tested. There is also a Translate , which translates the value of a property into a string. If you do not specify it, then ToString () will be used when outputting to the log.
In addition to the boolean flags descriptor, the library has a predefined integer counter descriptor (properties that end with a Count). It is very similar to the handle parsed above, so I will not bore the reader with his analysis.
Up to this point I have shown examples on the specifications of the figures. But we still have document specifications. Let us imagine the following situation: at the preliminary demonstration of our system, it turned out that the program incorrectly parses the documents. Instead of Cyrillic, squares are displayed. To avoid such errors from now on, we will write a test. Let us complement the specification of the document where the error was found with a line listing the section names:
In the test code we add the property of the same name:
We could start testing, but it will crash with an error. The library does not know how to handle this type of property. None of the well-known library descriptors is appropriate - the name of the property does not match either the flag or the counter. You need to create a new descriptor. We will declare it in the test code - this will be enough for the library to find it:
The above descriptor is applicable to a property called SectionNames. Convert separates the line read from the specification by commas and removes the extreme spaces. Verify determines that a property passes validation when two collections of strings are equivalent — each element from the first collection is present in the second collection and vice versa. Translate is needed so that in case of failure of the check, a meaningful inscription appears in the log, and not the name of an anonymous type. Translate creates a string that indicates the number of elements in the collection and lists their values separated by commas.
The handle is typed, so that when it is filled in, IntelliScense will suggest the types of arguments, and the compiler will check the correctness of the operations.
If you later need to add a property with the same behavior, then in the NamePattern descriptor you can change to @ "\ w + Names". And then all properties ending in Names will use this descriptor.
Test data can be located anywhere - the library does not impose any restrictions. However, in practice, it turned out to be convenient to use two test data storage locations - in the specification itself, or in a separate file. In both cases, the files are stored in the test project DLL as Embedded Resource. This allows:
Convenient for testing objects for which initialization does not need a lot of data. A descriptor is declared in the specification class in which the Verify method is not specified. The value of the property is taken from the dictionary of the read properties of the SpecifiedProperties specification:
Suitable for cases where there is a lot of data, or when the test data is a document. It is convenient to accept the agreement whereby the names of the specification and test data files are the same. In this case, the test data lie in the adjacent, relative to the specifications, directory. Then the reading will be as follows:
The testing engine relies on the following conventions:
For the case of storing test data in separate files, the structure will be as follows (data, as well as specifications, are stored in an assembly as Embedded Resource):
In specifications:
I applied the above approach to integration testing in my latest project and was pleased with the results. Tests based on specifications are easier to read and maintain compared to similar tests implemented as unit tests.
In fact, specifications are files written in Mini DSL (Domain Specific Language). The library is the engine of this language and will define the API for interacting with the code under test. In the general purpose language (C #), testing by specification can also be written, but readability will suffer from this and the costs of supporting tests will increase.
I think in the future I will add to the library the ability to indicate how long a particular property has the right to calculate. Then it will be possible to limit the time allotted for testing and check the SLA (System Level Agreements) agreements.
The specification is a text file describing what to test in the test data. It indicates what results the program should receive. The test code finds the real, calculated on the live code results. And the test engine performs verification of the specification and the calculated results.
This approach allows you to declaratively create tests. Specifications are easy to read and supplement when requirements change. The test code is compact. It is easy to maintain and expand.
')
The article describes the principles of the engine for testing specifications and provides examples of use. The engine itself is attached to the article. It can be considered a small library for integration testing.
Library code
Source code - stalker98.narod.ru/TestBySpecification.rar
The code is commented in detail, so if after reading the article some points remain unclear, the details can be viewed in the implementation. By and large, the library consists of only 4 classes, which is not much. The testing engine is in Utils.Common.Tests.
For testing, I use the 2008 studio test framework, but can be transferred to NUnit. To do this, you need to correct the test call in Test.Domain.Documents.DocumentTester.cs and Test.Domain.Polygons.PolygonTester.cs.
Subject area for examples
Let the task is to write a document converter from one format to another. The conversion is tricky, with a lot of mathematical calculations. The most difficult part of the transformation is the processing of geometric shapes described in the documents. The customer handed over a set of standard documents that he needs to translate into another format.
The development is still far away. Already written code is covered by unit-tests, each of which tests its module in isolation from the rest of the program. But there comes a time when it is required to make sure that the written code will work on real data. Classes of the program in this case should be used together, without isolation from each other. It is necessary to conduct integration testing.
The testing approach is the same - for each sample document sent by the customer, we will create a specification, where we write down the significant results that our program will receive when converting. From the sent documents we will choose various figures and also we will write for them the specifications. In the specifications of the figures we will write down the results that our program should come to when processing a particular figure.
Specification file
The specification is a text file that describes what to test in the test data. The file consists of:
- Comments at the beginning of the file. Comments can consist of any characters except $.
- Checked properties. The name of each property begins with a $ character. The end of the property value is determined either by the beginning of the next property, or by the end of the file.
4 3 |
Test code
To check the compliance of properties from the specification and real (calculated) property values, you need to know how to find these real property values. For this, a heir of the Specification class is created and get accessors for properties are written in it, with the same names as in the specification:
public class PolygonSpecification : Specification
{
/// <summary>
///
/// </summary>
private int VerticeCount
{
get { return Polygon.Vertices.Count( ); }
}
/// <summary>
///
/// </summary>
private bool IsRhombus
{
get { return Polygon.IsRhombus; }
}
/// <summary>
///
/// </summary>
private bool HasSelfIntersections
{
get { return Polygon.SelfIntersections.Any( ); }
}
// ...
}Property type must match the name. For example, properties that begin with Is, Has, or Are are considered by the library as Boolean flags. And properties ending in Count are integers. The mechanism of matching the name of a property to its type will be described below.
As you can see, the test code is minimized.
Challenge tests
[ TestClass ( )]
public class PolygonTester : Engine < PolygonSpecification >
{
/// <summary>
/// PolygonSpecification
/// </summary>
[ TestMethod ( )]
public void Polygon_AllSpecifications( )
{
Assert .IsTrue( TestSpecifications( ), FailedSpecInfo );
}
/// <summary>
///
/// </summary>
[ TestMethod ( )]
public void Polygon_DebugSpecification( )
{
Assert .IsTrue( TestSpecification( "" ), FailedSpecInfo );
}
}Launch testing in the studio Ctrl + R, A:
Test output
When testing, a log of operations is conducted (it is also a test output). What specifications and properties have been written to the log, what the test result is, how long the testing took. If a property has not been tested, this is indicated separately. Test output can be viewed by clicking on the test in the Test Results window.
I wrote 5 more specifications for various shapes. When testing, it turned out that the IsRhombus property (a boolean flag - whether the figure is a diamond) in the specifications does not match the value that is in the program. Obviously, an error was made in the method that determines whether the figure is a diamond. The log looks like this:
PolygonTester |
The mechanism for determining properties
The attentive reader at the moment should not understand two questions - where do the test data come from and what to do if you need to test something other than the equality of two bool or int. On both questions the answer lies in the mechanism for determining properties.
Each property belongs to one specific type. Moreover, the type here means more than just the .NET type. It is rather .NET type plus behavior when testing. The behavior is described by a special object - the property descriptor PropertyDescriptor <T>. The descriptor defines the convention adopted for property names, the conversion from string to property value and vice versa, as well as the criterion that determines whether the property passed the test or not.
I will cite as an example the library descriptor for Boolean flags:
protected PropertyDescriptor < bool > FlagProperty = new PropertyDescriptor < bool >
{
NamePattern = @"(Is|Has|Are)\w+" ,
Convert = ( value ) => bool .Parse( value ),
Verify = ( specified, actual ) => specified == actual,
};The descriptor specifies the regular expression NamePattern , which specifies a convention for naming properties to which the descriptor applies. In this case, these are all properties with names that begin with Is, Has, or Are. Convert sets the function to convert from string to property value. Verify defines the criteria for passing the test. In this case, it is a simple test for equality. That is, if a Boolean value in the specification is equal to the calculated value, then it is considered that the property passed the test. If Verify is omitted, the property will be read, but will not be tested. There is also a Translate , which translates the value of a property into a string. If you do not specify it, then ToString () will be used when outputting to the log.
In addition to the boolean flags descriptor, the library has a predefined integer counter descriptor (properties that end with a Count). It is very similar to the handle parsed above, so I will not bore the reader with his analysis.
Specification extension new type
Up to this point I have shown examples on the specifications of the figures. But we still have document specifications. Let us imagine the following situation: at the preliminary demonstration of our system, it turned out that the program incorrectly parses the documents. Instead of Cyrillic, squares are displayed. To avoid such errors from now on, we will write a test. Let us complement the specification of the document where the error was found with a line listing the section names:
$SectionNames = , , |
/// <summary>
///
/// </summary>
private IEnumerable < string > SectionNames
{
get { return Document.Sections.Select( s => s.Name ); }
}We could start testing, but it will crash with an error. The library does not know how to handle this type of property. None of the well-known library descriptors is appropriate - the name of the property does not match either the flag or the counter. You need to create a new descriptor. We will declare it in the test code - this will be enough for the library to find it:
/// <summary>
///
/// </summary>
protected PropertyDescriptor < IEnumerable < string >> SectionNamesProperty = new PropertyDescriptor < IEnumerable < string >>
{
NamePattern = @"SectionNames" ,
Convert = ( text ) => text.Split( ',' ).Select( n => n.Trim( ) ),
Verify = ( specified, actual ) =>
specified.Count( ) == actual.Count( ) &&
specified.Intersect( actual ).Count( ) == actual.Count( ),
Translate = ( value ) => string .Format( "[{0}]: {1}" ,
value .Count( ),
string .Join( ", " , value .ToArray( ) ) ),
};The above descriptor is applicable to a property called SectionNames. Convert separates the line read from the specification by commas and removes the extreme spaces. Verify determines that a property passes validation when two collections of strings are equivalent — each element from the first collection is present in the second collection and vice versa. Translate is needed so that in case of failure of the check, a meaningful inscription appears in the log, and not the name of an anonymous type. Translate creates a string that indicates the number of elements in the collection and lists their values separated by commas.
The handle is typed, so that when it is filled in, IntelliScense will suggest the types of arguments, and the compiler will check the correctness of the operations.
If you later need to add a property with the same behavior, then in the NamePattern descriptor you can change to @ "\ w + Names". And then all properties ending in Names will use this descriptor.
Read test data
Test data can be located anywhere - the library does not impose any restrictions. However, in practice, it turned out to be convenient to use two test data storage locations - in the specification itself, or in a separate file. In both cases, the files are stored in the test project DLL as Embedded Resource. This allows:
- Include integration tests along with all data in the version control system.
- Always have on hand specifications and test data when working in a studio.
1) Test data is sewn into specification
Convenient for testing objects for which initialization does not need a lot of data. A descriptor is declared in the specification class in which the Verify method is not specified. The value of the property is taken from the dictionary of the read properties of the SpecifiedProperties specification:
/// <summary>
/// ,
/// </summary>
PropertyDescriptor < IEnumerable < Vector >> VerticesProperty = new PropertyDescriptor < IEnumerable < Vector >>
{
NamePattern = "Vertices" ,
Convert = ( text ) => Polygon .ParseVertices( text ),
};
/// <summary>
///
/// </summary>
public Polygon Polygon
{
get { return new Polygon ( ( IEnumerable < Vector >) SpecifiedProperties[ "Vertices" ] ); }
}2) Test files in external file
Suitable for cases where there is a lot of data, or when the test data is a document. It is convenient to accept the agreement whereby the names of the specification and test data files are the same. In this case, the test data lie in the adjacent, relative to the specifications, directory. Then the reading will be as follows:
/// <summary>
///
/// </summary>
public Document Document
{
get
{
var assembly = Assembly .GetExecutingAssembly( );
var resourceName = string .Format(
"{0}.Documents.Data.{1}.txt" ,
assembly.GetName( ).Name, Name );
var stream = assembly.GetManifestResourceStream( resourceName );
var text = new StreamReader ( stream ).ReadToEnd( );
return Document .CreateFromText( text );
}
}Accepted Agreements
The testing engine relies on the following conventions:
- For each type of specification, a separate folder is created in the test project.
- In this folder, an inheritor from the Specification class is created.
- Specification files are added to the Specs subfolder as Embedded Resource.
- The specification of the file extension must be .spec
- Specifications are searched for in the same build from which the testing was launched. But if required, the assembly can be specified explicitly in Engine.Assembly.
For the case of storing test data in separate files, the structure will be as follows (data, as well as specifications, are stored in an assembly as Embedded Resource):
In specifications:
- Property names must match the calculated properties in the test code.
- Each read property must correspond to exactly one descriptor.
- The test code should not have an assumption in which order the properties are tested. Tests should not affect each other.
Conclusion
I applied the above approach to integration testing in my latest project and was pleased with the results. Tests based on specifications are easier to read and maintain compared to similar tests implemented as unit tests.
In fact, specifications are files written in Mini DSL (Domain Specific Language). The library is the engine of this language and will define the API for interacting with the code under test. In the general purpose language (C #), testing by specification can also be written, but readability will suffer from this and the costs of supporting tests will increase.
I think in the future I will add to the library the ability to indicate how long a particular property has the right to calculate. Then it will be possible to limit the time allotted for testing and check the SLA (System Level Agreements) agreements.
Source: https://habr.com/ru/post/81848/
All Articles