Code Coverage - I want to believe

The developer must know his tools! Knowledge of tools increases productivity, efficiency, productivity , the potency of the developer! I can not program without R #!

Such phrases can be heard from completely different people: fanatics of development, vendors of various utilities, users of convenient tools. My manager also hears them when I want to experiment with something new.

However, the instruction to the tool usually does not contain the section "Contraindications", does not indicate the situation when you should NOT use the utility. Meanwhile, a similar section could save tons of time for unsuccessful experiments.
')
Today, I rush stones to Code Coverage (CC). Quite a useful metric, under which are several poorly documented rakes.

"There is a lie, there is a blatant lie
there are statistics. "
The creators of SonarQube understand this perfectly well, it’s not for nothing that SonarQube has a dozen CC metrics . I will interrupt statistics using CC from DevExpress, there is only one metric.

Problem 1. Typical test wrong. Test the method with a bunch of argument checks:

public static Hash Example1_IfThen(int arg) { if(arg == 0) { throw new ArgumentException("0 argument"); } if (arg == 1) { throw new ArgumentException("1 argument"); } if (arg == 2) { throw new ArgumentException("2 argument"); } return new Hash(new Hash(arg + 42) + 13); } ... [TestMethod, ExpectedException(typeof(ArgumentException))] public void Example1_IfThen_0() { Program.Example1_IfThen(0); } [TestMethod, ExpectedException(typeof(ArgumentException))] public void Example1_IfThen_2() { Program.Example1_IfThen(2); } [TestMethod, ExpectedException(typeof(ArgumentException))] public void Example1_IfThen_1() { Program.Example1_IfThen(1); } 

The method is covered with tests for 83%, which is usually enough for an auto-build . Technically, there is nothing to argue about, most of the code is covered with tests, but the main scenario is not affected by tests. Tests covered the simplest part of the code, not the most important.

Problem 2. We measure the actual code, instead of the necessary.

  public static int Example2_IncompleteLogic(IEnumerable<int> arg) { return arg .Where(elem => elem != 2) .Where(elem => elem != 3) .Count(); } ... [TestMethod] public void OneTestToCoverAll() { // Arrange - collection with one element '4' var arg = new List<int> { 4 }; // Act var result = Program.Example2_IncompleteLogic(arg); // Assert Assert.AreEqual(1, result); } 

The test method does not contain a check for a null argument, but the coverage is 100%. Sometimes people forget: Code Coverage is a code coverage metric, not a requirement closure metric; if the method lacks logic (the method is not complicated enough to solve its problem) - CC will not show it.

100% coverage does not guarantee the performance of the program . Bringing to the point of absurdity: the empty method is elementarily covered at 100%. The nonempty method is covered by 100% tests without Assertions.

Problem 3. Optimism. A slightly different manifestation of the previous problem. As you can see, one test covers 100% of the code. Let's try to rewrite our method, getting rid of LINQ (to improve performance).

 var result = 0; foreach(var elem in arg) { if(elem == 2 || elem == 3) { continue; } else { result++; } } return result; 

We get only 73% coverage. The functionality has not changed, the metric has fallen. Not only that 100% coverage does not guarantee the performance of the program, these 100% can be fake . Conclusion: LINQ - g ** but the results of the CC may be too high, try to check the coverage in the editor.

A side observation: in this case, we can simply have a joint in the technical implementation; in theory, the anonymous method elem => elem! = 2 can be replaced by elem => if (elem! = 2) return true else return false; That will fix the coverage of the original method to 73%. True, such an approach would require complicating the now convenient UI.

Corollary: The tool used may not have all the desired functionality. A trivial thing, no less from that true.

Problem 4. Transfer of responsibility.

 public static void MakeEverythingWell() { OuterLib.MakeEverythingWell(); } 

The method is 100% covered with one test. At the same time, the OuterLib coverage of the library lies on the conscience of the person who added it. Or updated. About three years ago, before the introduction of CC. Before firing.
It is necessary to state the fact again: not only that 100% of the coverage does not guarantee the operation of the program, these 100% can be fake .

In addition to purely code points, there are several complaints about the processing of CC results.

Claim 0, known to all. 100% coverage. No 100% coverage - no build approval. The problem is that it is relatively easy to get the first percentages of coverage, but the last ones ... Especially when part of the code is generated. Or unattainable (since it was created for Vasya, who will use it in two days). Or just theoretically achievable, and an example to select / calculate a couple of weeks (this happens when working with mathematics). In short, most teams (of those who generally integrate CC into CI) stop at 60/70/80 percent of the required coverage.

Claim 1, controversial. Covering dead code. In my memory, a similar problem was most clearly manifested during the test by Mirand's colleagues from PVS. The comments are quite emotional, but some of the controversies concerned the dead code: some of the diagnostics found pointed to (abandoned) plugins, but not to the core .

The question arises: Is CodeCoverage needed for a dead code ? On the one hand, the dead code is a problem, and drawing attention to it is welcome. On the other hand, the dead code does not affect production, so is it worth allowing it to influence the CC metric?

Claim 2. The importance of the code. Expansion of the problem 1. In my project there are two remarkable controllers: “payment” and “negotiation”. "Payment" is critical for the client, and I fully agree with the requirement of "80% coverage", "negotiation" is used by 1.5 anonymus. In year. And it has not changed for two years. Question: why write tests to half-dead functionality? Just to get 80% badge auto approval ?

Claim 3, impossible. Metric as achivka. This is when no one checks what is covered. Remember the bikes about paying for the lines of the code? I have heard about people who created unnecessary code for better coverage.

Claim 4. Metrics "for free." When management throws off the requirement to “cover the code by 80%,” and the developers meekly agree. The project at the same time - disposable. Or a prototype. Or deadline on the nose. Or there is a hefty pasta legacy monster without a single test.

Code coverage tests takes time! If the coverage is also measured - time for tests may increase (although it may fall). So if the team did not have time to deliver the project on time, but it reached 80% of the coverage - the fault can be shared between management and developers. The question of the line of division of guilt should not be raised, for holivar.

At the end. I note once again: SS is a useful metric, albeit with surprises. It really helps with the control of the code, if there is no blind desire for numbers in the reports.