How static analysis can complement unit testing with the example of NUnit

Quite often, when discussing static analysis tools for C # projects, programmers write that this is not necessary, because with the help of unit testing they catch most of the errors. I decided to check how well one of the most famous unit test frameworks, NUnit, was tested, and see if our analyzer finds something there.

Introduction

NUnit is a popular unit testing library for .NET projects ported from Java to C #. The source code is open and available on the project website http://www.nunit.org/ .

It is worth noting that JUnit - the project from which NUnit was ported, was created by such famous programmers as Erich Gamma - one of the authors of the book about object-oriented design patterns, and Kent Beck - the creator of development methodologies through testing and extreme programming. I remember once reading his book Test Driven Development By Example, where he talks about developing through testing using the example of creating a test framework similar to JUnit, following all his methodologies. That is, there is no doubt that JUnit and NUnit are developed in the best traditions of unit testing, as Kent Beck’s quotation on NUnit site says: "... a great example of idiomatic design. Most of those who have ported xUnit simply transferred the version for Smalltalk or Java. The same thing we did with NUnit for the first time. This new version of NUnit is what it should have been if it had been written in C # from the beginning. "

I looked at the source code of NUnit - there are a lot of tests, there is a feeling that everything that is possible has been tested. Considering the excellent design and the fact that NUnit has been used by thousands of developers over the years, I thought that PVS-Studio would not find a single error in it. But no, the error was found.
')

About the error found

V3093 diagnostics worked that sometimes programmers instead of && and || use the operators & and |. A problem may arise when it is important that the right side of the expression is not executed under certain conditions. Let's see how this error looks in NUnit.

public class SubPathConstraint : PathConstraint { protected override bool Matches(string actual) { return actual != null & IsSubPath(Canonicalize(expected), Canonicalize(actual)); } } public abstract class PathConstraint : StringConstraint { protected string Canonicalize(string path) { if (Path.DirectorySeparatorChar != Path.AltDirectorySeparatorChar) path = path.Replace(Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar); .... } } 

Even if null is entered as the actual parameter in the Matches method, the right side of the & operator will still be calculated, which means the Canonicalize method will be called. If you look at its definition, you can see that in it the value of the path parameter is no longer checked for null , but it immediately calls the Replace method, where a potential NullReferenceException is possible. Let's try to reproduce the problem. For this, I wrote a simple unit test:

 [Test] public void Test1() { Assert.That(@"C:\Folder1\Folder2", Is.SubPathOf(null)); } 

Run and see the result:


So it is: NUnit fell from a NullReferenceException . Even in such a well-tested product as NUnit, the PVS-Studio static analyzer could find a real error. I note that it was no more difficult for me to do this than to write a unit test — you need to select a project check from the menu and analyze the grid with the results.

Conclusion

Unit tests and static analysis are not exclusive, but complementary software development techniques [ 1 ]. Download the PVS-Studio static analyzer and see if there are any errors that were not found by the unit tests.

Additional links

1. Andrey Karpov. How static analysis complements TDD .
2. Ilya Ivanov. One interesting bug in Lucene.Net .

If you want to share this article with an English-speaking audience, then please use the link to the translation: Alexander Chibisov. Complementing Unit Testing with Static Analysis, with NUnit as an Example .