Fixie - testing by agreement

Some time ago, I got a tweet that my acquaintance began to use the new test open source framework Fixie and was very pleased with it. So, I even decided to fix all the tests in my project for a new engine. After this, I simply could not stand aside and did not even look at what kind of animal it was and why it pleases others.

Next, I want to provide you with an overview of the framework, its capabilities, and to understand whether this is a new word in testing, if you look at it or you can go by and forget.

As stated on the site itself, Fixie is Conventional Testing for .NET. Those. testing by agreement. Agreements here means what we are accustomed to in general - all operations are carried out on the basis of an “oral” agreement, a gentleman's naming agreement. The closest example is scaffolding. This is when we agreed, for example, that test classes contain the word Test, or that test classes should be public and not return anything. Then such classes will be recognized as test. And no more attributes and all that stuff. Just classes and methods.
')
At first glance, it all looks good and even happy. It turns out that it is only necessary to correctly name the methods and classes and there will be happiness. At the same time, you can train the team to call test classes as necessary, rather than an arbitrary combination of words, somehow related to the topic of the class under test.

Installation and first test

By default, Fixie is configured so that the test classes are everything that ends in Tests . Test methods - everything inside these classes does not return values. Those. theoretically and, in fact, practically, such code will already be recognized as a test:

public class SuperHeroTests { public void NameShouldBeFilled() { var superHero = new SuperHero(); superHero.Name .Should() .NotBeEmpty(); } } 

Before proceeding further, it is worthwhile to say a few words about the installation, and what is necessary for the work of Fixie.



The nuget command to install the framework itself. But that’s not the end of it. The framework does not provide a built-in method for writing test data in a test. To use them, you need to use one of the third-party solutions:

• FluentAssertions
• Should
• Shouldly
• Something different from this topic

This is still set using NuGet. In the example above, the FluentAssertion package is used, since I personally like it more and there is essentially no big difference compared to other options.

That's all, and you can happily and fun to write code. If you just started your way in development and testing in particular. If you are an experienced person, there are many questions about how this thing is configured and how convenient it is to drive tests.

Integration

By the way about running tests. The author honestly admits that he had to run tests from the console and therefore he did integrate with the studio last. There is a plugin for ReSharper, but it is for versions from 8.1 to 8.3, i.e. With the new version you are a proletarian. For the sake of tests, I did not want to roll back to version 8, so I can’t say how comfortable it is.

Integration with the studio is performed at the level of detecting tests and running them. Those. There will be no highlighting in the editor. There are in view of special marks of tests.



Here, in my opinion, lies the potential place of error . Ochepyatki. It is not so difficult to be sealed in the word “Tests”, which will lead to the omission of tests. Visually, they do not stand out. For the sake of justice, I must say that such a situation is rare and unlikely if you use the right tools of the studio and run not all the tests, but, for example, only those that have not been run before.



In general, it is possible so far to refer to the youth of the framework and relative uncertainty and, as a result, the absence of many tools.

In order not to return twice to the topic of detecting tests and generating results, the documentation informs us that Fixie can generate reports in the NUnit and xUnit style, which will make life easier for those who normally have a CI built.

Tuning

The main strength of the framework is the ability to flexibly customize how your tests will be identified, how to run and how to be validated, and so on. By default, nothing comes with the framework. However, there are many different examples in the repository .

To make things more interesting, let's dig a little bit in how to customize the framework for your needs. For example, how to say which other classes should be identified as test classes.

All settings occur in the constructor class inherited from Convention.

 public class CustomConvention : Convention { public CustomConvention () { } } 

By default, the settings look something like this:

 public class DefaultConvention : Convention { public DefaultConvention () { Classes .NameEndsWith("Tests"); Methods .Where(method => method.IsVoid()); } } 

The code says that test classes must end with Tests and the test method will be all that returns null. In general, fit.

I try to adhere to the principle when the test class begins with the word When, and the tests with the words Then . It turns out quite a slim picture in the testranner and when writing tests you already know what you are testing, i.e. you only need to think about the effects for the test. An added bonus is that test classes are short and the responsibility between tests is not mixed.

Naturally, any rule only indicates direction, and is not a dogma. "A Foolish Consistency is the Hobgoblin of Little Minds" - so you should always be guided by common sense.



Looking at my list of tests, I see that not all classes end with the word Tests or begin with When .

There is no strict naming structure in tests either, in the sense that tests must begin with the word Should, for example. But in my opinion, reading tests perfectly understand what is happening there. I say this based on experience. This project has been simulating for 3-4 years already with varying success, and each time I quickly and successfully recall what has been done and what needs to be done. It's just some outlet when there is too much paperwork.

In this naming mode, I’m scared (and laziness) to specify keywords by which test classes will be defined. In addition, the approach When ... Then ... in practice means that there is a configuration method that runs at the beginning of each test and in the test itself either the results are checked or we influence the created object somehow. Those. it will be necessary to explicitly mark, or specify the configuration method (SetUp).

For implementation, I can explicitly invoke the tuning method in each test. For example, like this:

 public class SuperHeroTests { public void NameShouldBeFilled() { SetUpEnvironment(); var superHero = new SuperHero(); superHero.Name .Should() .NotBeEmpty(); } } 

Of course, in real life, you need to write something more informative than SetUpEnvironment (), but the idea is clear. With such an implementation, there will be class variables that will save their values ​​between tests - which can easily lead to dependent tests if I forget to prescribe an initialization string in some test.

Fixie offers a solution for this situation. Here it is:

 public class DefaultConvention : Convention { public DefaultConvention () { Classes .NameEndsWith("Tests"); Methods .Where(method => method.IsVoid()); CaseExecution .Wrap<HeroUniverseSetup>(); } } public class HeroUniverseSetup : CaseBehavior { public void Execute(Case context, Action next) { //  } } 

Those. it is necessary to implement the successor of CaseBehavior and register there everything that is necessary. Documentation says implementation

• CaseBehavior = [SetUp] / [TearDown]
• FixtureExecution = [FixtureSetUp] / [FixtureTearDown]

Well, but even so, in this approach, I see that I need to create many variants of DefaultConvention and prescribe initialization logic far from the test, while starting you will not even know that such a context exists.

It is easy to demonstrate. Let DefaultConvention be as in the example above, then when you run the class

 public class SuperHeroTests { public void NameShouldBeFilled() { var superHero = new SuperHero(); superHero.Name .Should() .NotBeEmpty(); } } 

Nothing tells me that there is SetUp / TearDown.



MAGIC!!! In general, it is interesting, but in this case, not. I would not want to imagine such magic on the project. This is completely contrary to the fact that tests should be transparent. Even if I write an extension for CaseBehavior in the same class , this is not a solution, since it will not be very obvious where I will look for a class in which all this will be configured. Doing constant save-paste is also not an option.

Compare:

 public class DefaultConvention : Convention { public DefaultConvention () { Classes .InTheSameNamespaceAs(typeof(DefaultConvention)) .NameEndsWith("Tests"); Methods .Where(method => method.IsVoid()); CaseExecution .Wrap<HeroUniverseSetup>(); } } public class HeroUniverseSetup : CaseBehavior { public void Execute(Case context, Action next) { //  } } public class SuperHeroTests { public void NameShouldBeFilled() { var superHero = new SuperHero(); superHero.Name .Should() .NotBeEmpty(); } } 

Vs:

 [TestFixture] public class SuperHeroTests { [SetUp] public void Init() { //  } [Test] public void NameShouldBeFilled() { var superHero = new SuperHero(); superHero.Name .Should() .NotBeEmpty(); } } 

The lines of code will somehow be smaller in the case of NUnit, and SetUp from the class will not be lost in the project.

Maybe I am biased and write tests incorrectly, but ... I'm not sure about that. I have not yet met a lot of articles that SetUp is evil. Those. everything can be brought to the point of absurdity and initialization in SetUp of half the project, but this is a different matter.



However, in this case there is a solution. You can use handwritten attributes to determine the desired parts of the test, tests, and classes. Those. You can fully emulate NUnit, xUnit in your project.

With the help of attributes you can do

• category support
• parameterized tests
• strict tests
• all that comes to mind

I will give here a small example of creating categories and launching them.

 public class CustomConvention : Convention { public CustomConvention() { var desiredCategories = Options["include"].ToArray(); var shouldRunAll = !desiredCategories.Any(); Classes .InTheSameNamespaceAs(typeof(CustomConvention)) .NameEndsWith("Tests"); Methods .Where(method => method.IsVoid()) .Where(method => shouldRunAll || MethodHasAnyDesiredCategory(method, desiredCategories)); if (!shouldRunAll) { Console.WriteLine("Categories: " + string.Join(", ", desiredCategories)); Console.WriteLine(); } } static bool MethodHasAnyDesiredCategory(MethodInfo method, string[] desiredCategories) { return Categories(method).Any(testCategory => desiredCategories.Contains(testCategory.Name)); } static CategoryAttribute[] Categories(MethodInfo method) { return method.GetCustomAttributes<CategoryAttribute>(true).ToArray(); } } 

And run tests depending on the category:
Fixie.Console.exe path / to / your / test / project.dll --include CategoryA

My decision

You can say a lot more about Fixie, give examples of solving various problems, but the first impression is already there. Therefore it is worth rounding out.

It turns out that Fixie is a metaframe for testing. You are free to build any rules and opportunities for testing, while they are built quite simply, to be honest. The only question is expediency. Is it necessary to do it all? A little scares away, of course, the lack of support for R #, and the fact that I don’t see that the test is really a test, and it was recognized by the framework. In production, I would not use it, but for home use and as a promising tool - Fixie is very interesting . At least, I will definitely remember it if there is some interesting and specific testing task that will be difficult to solve using standard NUnit tools.