Comparison of code analyzers: CppCat, Cppcheck, PVS-Studio, Visual Studio
Authors: Evgeny Ryzhkov ( EvgeniyRyzhkov ), Andrey Karpov ( Andrey2008 ), Pavel Eremeev ( Paull ), Svyatoslav Razmyslov ( svyatoslavmc ).
We conducted a detailed comparison of four C / C ++ code analyzers: CppCat, Cppcheck, PVS-Studio and a static analyzer built into Visual Studio 2013. This is a serious study that we spent about 170 man-hours and which, in our opinion , well reflects the picture that exists at the moment.
More than ten open-source projects have been tested for research.
The recommended level of detail for diagnostic messages has been set:
')
The comparison methodology is described in detail in the article “How we compared code analyzers CppCat, Cppcheck, PVS-Studio, Visual Studio” ( in Russian , in English ). It will also be possible to find the names of checked projects, which diagnostic messages were selected, examples of errors found, and so on. It also describes the reason why it was not possible to provide information on the number of false positives.
The results of the comparison of CppCat, Cppcheck, PVS-Studio and Visual Studio analyzers are presented in the following three tables:
Table 1 - the Number of detected real problems, in pieces
Table 2 - The number of detected real problems, in percent
Table 3 - Analyzers operating time, in minutes
We do not consider analysis time to be an essential comparison parameter. But about the speed of work they regularly ask questions, so we included time measurements in this article.
As you can see, the fastest is Cppcheck. But keep in mind that this is the total time. On individual projects, other analyzers are faster than Cppcheck. It all depends on the project.
CppCat actually works a little bit faster than PVS-Studio, since it performs fewer checks. But the difference is so insignificant (tens of seconds) that it makes no sense to mark it in the table.
In total, our team identified in projects 965 unique code fragments that deserve close attention.
The table shows that CppCat and PVS-Studio showed identical results. The fact is that with the recommended settings, the PVS-Studio analyzer generates the same messages as CppCat.
The total CppCat analyzer, like PVS-Studio, detects 742 dangerous places in the code of projects. Cppcheck - 193, Visual Studio - 116.
It can be said that the CppCat and PVS-Studio analyzers detect 4 times more errors than Cppcheck and 6 times more errors than the analyzer from Visual Studio.
The CppCat and PVS-Studio analyzers proved to be clear leaders in the number of defects found and potential defects.
We conducted a detailed comparison of four C / C ++ code analyzers: CppCat, Cppcheck, PVS-Studio and a static analyzer built into Visual Studio 2013. This is a serious study that we spent about 170 man-hours and which, in our opinion , well reflects the picture that exists at the moment.
About comparison
More than ten open-source projects have been tested for research.
The recommended level of detail for diagnostic messages has been set:
- CppCat: all diagnostics (no levels);
- Cppcheck: Errors and Warnings;
- PVS-Studio: Levels 1 and 2 of general-purpose diagnostics;
- Visual Studio: Microsoft Native Recommended Rules.
')
The comparison methodology is described in detail in the article “How we compared code analyzers CppCat, Cppcheck, PVS-Studio, Visual Studio” ( in Russian , in English ). It will also be possible to find the names of checked projects, which diagnostic messages were selected, examples of errors found, and so on. It also describes the reason why it was not possible to provide information on the number of false positives.
Comparison results
The results of the comparison of CppCat, Cppcheck, PVS-Studio and Visual Studio analyzers are presented in the following three tables:
Time
We do not consider analysis time to be an essential comparison parameter. But about the speed of work they regularly ask questions, so we included time measurements in this article.
As you can see, the fastest is Cppcheck. But keep in mind that this is the total time. On individual projects, other analyzers are faster than Cppcheck. It all depends on the project.
CppCat actually works a little bit faster than PVS-Studio, since it performs fewer checks. But the difference is so insignificant (tens of seconds) that it makes no sense to mark it in the table.
Diagnostics
In total, our team identified in projects 965 unique code fragments that deserve close attention.
The table shows that CppCat and PVS-Studio showed identical results. The fact is that with the recommended settings, the PVS-Studio analyzer generates the same messages as CppCat.
The total CppCat analyzer, like PVS-Studio, detects 742 dangerous places in the code of projects. Cppcheck - 193, Visual Studio - 116.
It can be said that the CppCat and PVS-Studio analyzers detect 4 times more errors than Cppcheck and 6 times more errors than the analyzer from Visual Studio.
Conclusion
The CppCat and PVS-Studio analyzers proved to be clear leaders in the number of defects found and potential defects.
Additionally
- Evgeny Ryzhkov, Andrei Karpov, Pavel Yeremeyev, Svyatoslav Razmyslov. “How we compared code analyzers CppCat, Cppcheck, PVS-Studio, Visual Studio” ( in Russian , in English ).
Read the article and immediately want to ask?
Often our articles are asked the same questions. We collected the answers to them here: Answers to questions from readers of articles about PVS-Studio and CppCat, version 2014 . Please review the list.
| Unfortunately, we no longer develop or support the CppCat project. You can read about the reasons here . |
Source: https://habr.com/ru/post/215469/
All Articles