Viewing statistics on the number of errors in a project, or “Wow, graphics have appeared in PVS-Studio!”

This note will be of interest primarily to project managers and technical managers, whose teams use the PVS-Studio code analyzer. The tool has the ability to track the effectiveness of the use of a static analyzer in teams. Now you can prove in numbers to the boss that the analyzer bought for several thousand dollars brings real, visible benefits. But this article is not about ROI, do not be intimidated.

So, what's the problem with static analysis tools besides the fact that they are expensive? Not always the team that purchased the tool can boast that errors are corrected with its help. And we are more than anyone else interested in the effective use of our tool. We are not satisfied if the client simply buys a license and puts it on the shelf. After all, we have more than half of our clients extend licenses for the next year. Therefore, our task is to show the effectiveness of using our tool for those who decide to extend the license.
')
Therefore, in PVS-Studio 5.27, it became possible to plot the number of errors detected by the analyzer when checking a project. The idea of ​​this feature is very simple:

1. If you want to correct all the errors that PVS-Studio gives out, over time your schedule should come to zero.
2. If you are ready to put up with old mistakes, but rule all new ones, then your schedule should not grow much upwards.
3. Otherwise, the developers simply do not use PVS-Studio. Unfortunately, for us (we will not get a license renewal), and for you (you have wasted money on a license).

Before looking at specific graphics, just a couple of words about technology. More information about this feature can be found in the documentation .



Each time the entire code is checked (the Check Solution command), the analyzer saves in the% AppData% \ Roaming \ PVS-Studio \ Statistics folder a small XML file in which the number of messages is recorded. In the PVS-Studio menu there is an Analysis Statistics command, which opens the statistics display settings dialog. Choose a time interval, choose analyzers that are interesting to us and the importance of messages. Then, using the “Show in Excel” button, the Microsoft Excel installed on the machine opens in which the desired graph is plotted.

It makes no sense to talk long about this dialogue, everything is described in the documentation. So let's take a closer look at possible situations in the examples.

What does the graph look like when, after the introduction, all errors (both old and new) are corrected?

Consider the first example. The company bought PVS-Studio, starts to implement it. Let us be interested in the errors of the class General Analysis (GA), and to begin with, only the most important and medium importance. When you first start, PVS-Studio issues approximately 1800 messages. First, developers deal with the most obvious false positives. For example, disable them through suppression in macros. Rule more simple errors that are easy. Over time, the speed of working with messages slows down a bit. That is why the graph we have is not a straight line, but arched.

There are fewer and fewer errors, and finally sometime they become 0. Here is a graph describing this mode of operation of the command over PVS-Studio errors:



Figure 1 - First, minor errors are corrected, then more complicated and the most difficult ones remain at the end.

Everything, all analyzer triggers are worked out, 0 errors are issued. Can you relax? Not. After all, if you relax and further disable the analyzer (do not edit new errors, then over time the number of errors will start to grow. An example of such a graph is shown in Figure 2.



Figure 2 - After all errors were first corrected, the developers relaxed and stopped responding to the analyzer messages.

See this graph shows the increase in the number of messages? This is due to the fact that, bringing the number of positives to zero, the developers stopped correcting errors in the new code. Fortunately, the project manager remembered in time and noticed that mistakes do not rule. Therefore, after talking with the team in the project, it again became 0 errors. And so much and maintained daily.

An example of an incorrect conclusion about the results of the team based on the schedule

Okay, but what can you say on the schedule in Figure 3?



Figure 3 - An example of a graph with a double interpretation.

At first glance, we can say that the team is splintered, nothing rules and its number of errors grows. And quite strongly - from 2700 to 3700 for the period. But this conclusion can be made by a person who is not at all familiar with the project. If you know what people do, it becomes clear. A large amount of new code was added to the project (new project files were connected to the common .sln solution). And there is nothing wrong with such a schedule. After all, the number of projects further stabilizes, errors will not grow much, if the team begin to rule them on time.

This graph is given as an example of the fact that it is impossible to “interpret” statistics data, you still need to understand what the team is doing at one time or another.

Where do the steps come from?

Often, the correct form of the graph with a uniform operation resembles steps as in Figure 4.



Figure 4 - Step graph.

Such a stepped chart is the most correct one, because it means the uniform work of the error correction team. Guess where steps come from when the number of errors does not decrease? Of course this is the weekend.

What is the “only new” error chart?

Well, before that there were examples when the team worked on correcting all the errors that the code analyzer generated. But after all, the essence of the " mass error suppression " function is to say: “Okay, we know that we have a lot of PVS-Studio messages on the old code. But now we want to close our eyes on him and correct errors only in the new code. ” What will the error graph look like in this case? Figure 5 answers this question.



Figure 5 - Graph of the number of errors for only the new code.

Although at first glance the graph looks chaotic, this is exactly what a normal workflow should look like using a code analyzer. What is meant? Pay attention to the scale along the Y axis. The maximum value here is 8 - this is the maximum number of errors (analyzer responses) were added in one day. Say it a lot? But if we are talking about a team of 50 developers and 9 million lines of code, then you will agree, this is a very good result. But more importantly, they rule the next day.

What is the correct scenario of using PVS-Studio so that the “schedule was perfect”?

So, we understand how the "wrong" graphics look and how the "correct" ones look. But how should the development process be structured in order for the graphics of the PVS-Studio errors to be “correct”? What should a project manager or technical manager organize?

Here are the steps that will lead to the most effective result:

1. Set up daily verification of the code with PVS-Studio on the assembly server. This will help you control the error correction process and accumulate history, which is then easy to visualize.
2. For all developers, install PVS-Studio and enable the incremental analysis mode. In this mode, PVS-Studio keeps track of which files the developer rules on his own machine. And after their successful compilation automatically checks. If errors are found, they will of course be shown. The advantage of this mode is that the developer does not need to explicitly run the analyzer. And after seeing the message from him, he can immediately correct the code. In this case, the erroneous code does not even fall into the version control system.
3. If the erroneous code gets into the version control system, it will be detected during the night check. It is very important that the developers see the report on the results of the night launch in the morning. Then they can correct the mistakes. PVS-Studio automatically generates a report file with errors found in the .html format, which can be sent to interested parties.

If the team can enter this mode of using the code analyzer, then the implementation of the tool will not be in vain. And the question of extending the license for the next year will be solved by itself.