How not to present the results of the static analyzer
On the open spaces of the network is the blog " No Marketing Bullshit ". The unknown author of the blog seems to be one of our fans, as evidenced by a couple of notes on the topic of the PVS-Studio code analyzer. I decided to translate one of these notes. Just in case, I note that we have nothing to do with this blog. This is not our style. If we are going to write an advertising article, then we do not disguise, and we write - this is an advertising article.
In addition to the sad fact that it is impossible to try Coverity without going through the Coverity Sales Force Approval, the next big point that distinguishes Coverity from PVS-Studio is ... their promotional material.
Let's take a look at a typical project verification report using PVS-Studio. This one caught my eye first, so I link to it.
A typo ... Yes, I see. The risk of going beyond the boundary of the array ... Everything is clear here. A couple of other minor issues ... I see. No idea how this problem can affect the work of the program, but they are written about well enough and understandable for those who are interested.
')
One may wonder how the program can work with these errors .
Everything is quite simple. Defects that actually somehow manifest themselves have already been discovered earlier using other methods — good old debugging programs, unit tests, help from colleagues, etc. More efforts are required to find the remaining problems. It could be a clumsy dataset. This may be some unusual sequence of user actions. This may be an indication of a specific error. This can update the compiler or C ++ runtime.
Defects are defects. You can say “but you can't lie to the compiler,” but not all errors are the same. These little details can hang in the code base for years and then one day, someone runs the PVS-Studio analyzer on the source code and THIS IS YES! I HAVE THE SAME HUGE WHOLE BUMBLE OF BAGS! FUGAS ME IN EYE!
Of course, by itself, such a report is by and large useless - it is necessary that there be a developer who is familiar with the code base and can correct this defect. PVS-Studio reports are exactly what you need - they describe these shortcomings, providing some analysis, nothing superfluous.
Now let's take a look at how Coverity writes. Finding such a report, similar to what we have just seen, will be very difficult among the materials of Coverity. But once every five years, the guys from Coverity still issue an “Integrity Report”
The Integrity Report is a very colorful document where you can find words such as “mission”, “flawlessly” and “focus on innovation”. Not bad for a start - but at least the fact that these keywords are present in this document clearly show that there are already too many ads on the first three pages.
Moving on to Table A ... Hmm ... this table shows the size distribution of the projects. The use of the word “distribution” implies that the data collected has some statistics and is trustworthy. Even checking, say, 45 projects, it would be unreasonable to try to make such a table. They have the entire TWO project and more than 7 million lines of code. It's just incredible, I have no words.
The rest of the report is full of the same absolutely meaningless tables. No, well, of course it's cool that you can find 9.7654 problems per 30 square centimeters in the project. But as long as I do not try your program myself, I don’t care, since these numbers are no more than an indicator of 132% efficiency (a post five years ago, but still relevant ).
We turn to Appendix A. Tables 2 and 3 summarize all the problems, assigning them a category and degree of importance. And so, let's see ...
Problems with flow control. What is this? Is this when I forget to put a “break;” at the end of a “case” inside a “switch”? So you say that this is a medium degree of significance. Okay. How about the fact that the function "main ()" returns control immediately? This is a real problem in the flow of control and do not say that it is only average. Not all problems in the control flow are equivalent.
Dereferencing a NULL mid-level pointer, isn't it? Well, of course, my code here and there is dereferencing null pointers and every time, as it happens, users get a candy. Perhaps the authors of this Integrity Report had in mind dereferences of potentially null pointers when a pointer is dereferenced, without first checking whether it is zero or not. And the great news is that checking the pointer every time before it is dereferenced takes a lot of time. Again, not all dereferencing null pointers are the same.
The problem with error handling is a moderate degree. What is it? Check for code errors in Win32 API functions? Naturally, the fact that every time a program wants to open a file and simply continues reading without checking whether it was possible to open it or not will be a trifle for the user. No access to the folder? Let's pretend that we still saved the file. And so come down. Not all error handling problems are the same.
Problems in handling integers have an average level of significance. Of course, an integer overflow in calculating the size of the memory allocation is nonsense at all. Just let some memory be allocated, and we pretend that this is the correct size. Not all the problems of processing integers are the same.
Handling unprotected data has an average level of significance. What is this? In general, without a clue, but I still say that not all cases of processing unprotected data are similar.
Invalid expression - medium degree. Of course, put brackets where you want (where the left heel wants), what problems?
Concurrent access violation - medium level. Just think, you will rule these places until the end of your life.
API usage errors are middle level. Your code mistakenly forgets to specify the path and because of this, all the contents of Windows \ System32 are deleted. Sneeze on it.
The program hangs - the average degree. The program hangs only when running on a computer outside the Windows NT domain. You just run it in your work network, then you go to the trade show and your laptop turns into a heater with a monitor for a thousand dollars. What a trouble.
And I wonder why there is no category with a low level? Is it because the authors did not dare to name the flaw in low-level software because it already belongs to some category?
Will not work. It will not be possible to scatter several thousand defects in several categories and then assign a level of significance for them. It is simply impossible. If you are a software developer, you no doubt need to understand this. Otherwise - you should go to the nearest McDonalds - they have the item “Help Required” there.
In general, the Integrity Report is just a mess of numbers and diagrams. Its use is not something that is zero, but even a minus. Such a report will scare anyone who cares about the quality of their programs.
So, what is the difference between the PVS-Studio and Coverity advertising materials? That the first provide facts that can be understood and verified. And the latter simply frighten with their huge numbers, without giving a chance to check it somehow.
Because not everyone deserves a trial version of Coverity.
In addition to the sad fact that it is impossible to try Coverity without going through the Coverity Sales Force Approval, the next big point that distinguishes Coverity from PVS-Studio is ... their promotional material.
How could this be done
Let's take a look at a typical project verification report using PVS-Studio. This one caught my eye first, so I link to it.
A typo ... Yes, I see. The risk of going beyond the boundary of the array ... Everything is clear here. A couple of other minor issues ... I see. No idea how this problem can affect the work of the program, but they are written about well enough and understandable for those who are interested.
')
One may wonder how the program can work with these errors .
Everything is quite simple. Defects that actually somehow manifest themselves have already been discovered earlier using other methods — good old debugging programs, unit tests, help from colleagues, etc. More efforts are required to find the remaining problems. It could be a clumsy dataset. This may be some unusual sequence of user actions. This may be an indication of a specific error. This can update the compiler or C ++ runtime.
Defects are defects. You can say “but you can't lie to the compiler,” but not all errors are the same. These little details can hang in the code base for years and then one day, someone runs the PVS-Studio analyzer on the source code and THIS IS YES! I HAVE THE SAME HUGE WHOLE BUMBLE OF BAGS! FUGAS ME IN EYE!
Of course, by itself, such a report is by and large useless - it is necessary that there be a developer who is familiar with the code base and can correct this defect. PVS-Studio reports are exactly what you need - they describe these shortcomings, providing some analysis, nothing superfluous.
How not to do it
Now let's take a look at how Coverity writes. Finding such a report, similar to what we have just seen, will be very difficult among the materials of Coverity. But once every five years, the guys from Coverity still issue an “Integrity Report”
The Integrity Report is a very colorful document where you can find words such as “mission”, “flawlessly” and “focus on innovation”. Not bad for a start - but at least the fact that these keywords are present in this document clearly show that there are already too many ads on the first three pages.
Moving on to Table A ... Hmm ... this table shows the size distribution of the projects. The use of the word “distribution” implies that the data collected has some statistics and is trustworthy. Even checking, say, 45 projects, it would be unreasonable to try to make such a table. They have the entire TWO project and more than 7 million lines of code. It's just incredible, I have no words.
The rest of the report is full of the same absolutely meaningless tables. No, well, of course it's cool that you can find 9.7654 problems per 30 square centimeters in the project. But as long as I do not try your program myself, I don’t care, since these numbers are no more than an indicator of 132% efficiency (a post five years ago, but still relevant ).
We turn to Appendix A. Tables 2 and 3 summarize all the problems, assigning them a category and degree of importance. And so, let's see ...
Problems with flow control. What is this? Is this when I forget to put a “break;” at the end of a “case” inside a “switch”? So you say that this is a medium degree of significance. Okay. How about the fact that the function "main ()" returns control immediately? This is a real problem in the flow of control and do not say that it is only average. Not all problems in the control flow are equivalent.
Dereferencing a NULL mid-level pointer, isn't it? Well, of course, my code here and there is dereferencing null pointers and every time, as it happens, users get a candy. Perhaps the authors of this Integrity Report had in mind dereferences of potentially null pointers when a pointer is dereferenced, without first checking whether it is zero or not. And the great news is that checking the pointer every time before it is dereferenced takes a lot of time. Again, not all dereferencing null pointers are the same.
The problem with error handling is a moderate degree. What is it? Check for code errors in Win32 API functions? Naturally, the fact that every time a program wants to open a file and simply continues reading without checking whether it was possible to open it or not will be a trifle for the user. No access to the folder? Let's pretend that we still saved the file. And so come down. Not all error handling problems are the same.
Problems in handling integers have an average level of significance. Of course, an integer overflow in calculating the size of the memory allocation is nonsense at all. Just let some memory be allocated, and we pretend that this is the correct size. Not all the problems of processing integers are the same.
Handling unprotected data has an average level of significance. What is this? In general, without a clue, but I still say that not all cases of processing unprotected data are similar.
Invalid expression - medium degree. Of course, put brackets where you want (where the left heel wants), what problems?
Concurrent access violation - medium level. Just think, you will rule these places until the end of your life.
API usage errors are middle level. Your code mistakenly forgets to specify the path and because of this, all the contents of Windows \ System32 are deleted. Sneeze on it.
The program hangs - the average degree. The program hangs only when running on a computer outside the Windows NT domain. You just run it in your work network, then you go to the trade show and your laptop turns into a heater with a monitor for a thousand dollars. What a trouble.
And I wonder why there is no category with a low level? Is it because the authors did not dare to name the flaw in low-level software because it already belongs to some category?
Will not work. It will not be possible to scatter several thousand defects in several categories and then assign a level of significance for them. It is simply impossible. If you are a software developer, you no doubt need to understand this. Otherwise - you should go to the nearest McDonalds - they have the item “Help Required” there.
In general, the Integrity Report is just a mess of numbers and diagrams. Its use is not something that is zero, but even a minus. Such a report will scare anyone who cares about the quality of their programs.
Results
So, what is the difference between the PVS-Studio and Coverity advertising materials? That the first provide facts that can be understood and verified. And the latter simply frighten with their huge numbers, without giving a chance to check it somehow.
Because not everyone deserves a trial version of Coverity.
Source: https://habr.com/ru/post/273411/
All Articles