Intersection of PVS-Studio and Cppcheck
We were repeatedly asked the question of how strongly the diagnostics of our PVS-Studio analyzer and Cppcheck analyzer overlap. I decided to write a small article on this topic in order to quickly answer this question. If it is very short - it intersects weakly. Only 6% of the total number of errors are found by both analyzers. The article will explain how this number was received.
In the beginning, I wanted to draw a Venn diagram in the form of beautiful circles. But it turns out this is a whole task. Excel draw circles without considering their area. And the programs that draw the correct proportional diagrams are paid. So I limited myself to squares, for which I needed only a calculator for calculations, a pen with paper and the editor Paint.
Figure 1. Visual display of the number of errors found using the PVS-Studio and Cppcheck analyzer.
The area of squares is proportional to the number of errors found. The gray box represents the number of errors that both analyzers simultaneously find.
')
Total:
The data were obtained as follows. In March 2014, we conducted a large comparison of four code analyzers: PVS-Studio, CppCat, Cppcheck, Visual Studio:
The results have been heavily criticized by some of our readers. But we are sure that most of the criticism is connected with the fact that people have read brief conclusions, but have not read carefully the article describing the process of comparison itself.
Since the PVS-Studio analyzer showed itself much better than Cppcheck, some readers decided that we cheated. In fact, there is no deception. The PVS-Studio analyzer is indeed more powerful than Cppcheck. It is not clear that this is unexpected and suspicious. Commercial tools are usually better than free analogues. The high quality of the comparison analyzers, confirmed by the author himself Cppcheck. No matter if we publish an article where we give his letter about the comparison of analyzers, we will also answer a number of questions that were asked by readers after the publication of the article about comparison.
Let's return to the diagnostic intersection. As you can see, the intersection is rather weak, but this is not surprising. First of all, nobody wants to copy all diagnostics from another analyzer. The intersection occurs on the errors, the patterns of which are obvious and often occur. Thus, authors come to the idea of finding such errors independently.
Secondly, the era of static analyzers is just beginning. There is an incredible amount of error patterns that they can diagnose. Therefore, so few intersections. One analyzer relies on errors of one type, the other on others. It is clear that over time, the intersection will gradually increase. But there are so many patterns that this process will be slow. In addition, with the advent of C ++ 11 and C ++ 14, the field for activity only increases.
In the beginning, I wanted to draw a Venn diagram in the form of beautiful circles. But it turns out this is a whole task. Excel draw circles without considering their area. And the programs that draw the correct proportional diagrams are paid. So I limited myself to squares, for which I needed only a calculator for calculations, a pen with paper and the editor Paint.
Figure 1. Visual display of the number of errors found using the PVS-Studio and Cppcheck analyzer.
The area of squares is proportional to the number of errors found. The gray box represents the number of errors that both analyzers simultaneously find.
')
Total:
- Number of errors found with PVS-Studio: 742
- Cppcheck errors found: 193
- In total, the analyzers found 884 unique errors.
- The number of errors that are detected by both analyzers: 51
- Crossing is: 6%.
The data were obtained as follows. In March 2014, we conducted a large comparison of four code analyzers: PVS-Studio, CppCat, Cppcheck, Visual Studio:
- Brief comparison results ;
- Details on how and what was compared;
- For completely untrusting: Excel file with all errors found.
The results have been heavily criticized by some of our readers. But we are sure that most of the criticism is connected with the fact that people have read brief conclusions, but have not read carefully the article describing the process of comparison itself.
Since the PVS-Studio analyzer showed itself much better than Cppcheck, some readers decided that we cheated. In fact, there is no deception. The PVS-Studio analyzer is indeed more powerful than Cppcheck. It is not clear that this is unexpected and suspicious. Commercial tools are usually better than free analogues. The high quality of the comparison analyzers, confirmed by the author himself Cppcheck. No matter if we publish an article where we give his letter about the comparison of analyzers, we will also answer a number of questions that were asked by readers after the publication of the article about comparison.
Let's return to the diagnostic intersection. As you can see, the intersection is rather weak, but this is not surprising. First of all, nobody wants to copy all diagnostics from another analyzer. The intersection occurs on the errors, the patterns of which are obvious and often occur. Thus, authors come to the idea of finding such errors independently.
Secondly, the era of static analyzers is just beginning. There is an incredible amount of error patterns that they can diagnose. Therefore, so few intersections. One analyzer relies on errors of one type, the other on others. It is clear that over time, the intersection will gradually increase. But there are so many patterns that this process will be slow. In addition, with the advent of C ++ 11 and C ++ 14, the field for activity only increases.
| Unfortunately, we no longer develop or support the CppCat project. You can read about the reasons here . |
Source: https://habr.com/ru/post/236795/
All Articles