The real truth about comparing CodeSonar and PC-lint

Immediately I want to draw the attention of readers that this article is a translation and I, as the developer of the competing product PVS-Studio, will refrain from my own assessments of the situation.

Being recognized leaders and instigators of the static analysis industry, we (Gimpel Software) are flattered that other companies are guided by our product as a quality standard when developing their tools. Usually we do not consider it necessary to react in any way to publications about the results of comparisons of our analyzer with other products, however the “expert report” issued by Grammatech and devoted to the comparison of the CodeSonar and PC-lint tools turned out to be a very unpleasant exception. Instead of focusing on the merits of their product, the authors of this document resorted to false statements about PC-lint and distortion of facts regarding its technical capabilities, which was probably the result of pressure from the market, and we consider it our duty to respond to this lie, having told about the real state of affairs.

About PC-lint

PC-lint is an effective and respected among developers tool for static code analysis in C and C ++. It was created by Gimpel Software in 1985 and has been developing continuously ever since. During all these 30 years, PC-lint has been a leader in the industry, offering users innovative features, such as data tracking as they move between functions and modules of the program, strict type checking and dimensional analysis, and support for user-defined functions. PC-lint is trusted by tens of thousands of developers, technical control experts, testers and forensic experts; supports dozens of compilers and platforms and offers a range of additional options. PC-lint is used everywhere in virtually every industry, including areas with increased security requirements, such as medicine and automotive.

About the Grammatech Report

The expert report under the heading “How CodeSonar Compares To PC-lint (And Similar Tools)” (“CodeSonar Comparison with PC-lint (and other similar tools)”) is available on the Grammatech website. This document claims to be a reliable presentation of the comparison results of the CodeSonar and PC-lint tools (it also mentions other analyzers, but the main emphasis is on PC-lint), but in reality is nothing more than a set of self-serving, false statements that are not supported by real facts. and designed to mislead readers. Based on these false statements, the authors of the report are trying to present the CodeSonar product in a favorable light at the expense of PC-lint.
')

Retraction plan

In an attempt to blacken PC-lint, the authors of the document resort to certain tactical methods, namely:

• They build their reasoning on the assumption that the PC-lint hardware and functionality have practically not evolved during its existence.
• Intentionally misinterpret the tasks and capabilities of PC-lint.
• Make false claims about the diagnostic abilities of PC-lint.

Two sections of this article will be devoted to the examination and refutation of these statements. In the “Charges” section, we examine the key points of the report, most of which are not supported by any evidence, and provide the facts. In the section “Distortion of facts by examples,” we consider the examples of the code used in the report with errors that, according to the authors, cannot be detected using PC-lint, and show the real diagnostic messages of our tool with the results of the analysis of these fragments. The report also contains a number of valid criticisms - we will look at them in the appropriate section.

Charges

The Grammatech report offers a number of rather vague and extremely inaccurate statements, for example:

• Static analysis tools designed to find software errors in the source code have existed for several decades. First-generation analyzers, such as tools of the lint family, are now considered primitive. Such tools include both commercial products, for example, PC-lint, and open projects, for example, Cppcheck. In recent years, they have been superseded by more advanced tools, which include CodeSonar.

Although we agree that the original “lint” in Unix was very primitive, an attempt to attribute this same quality to our product based only on a similar name seems at best insincere. For over 30 years, Gimpel Software has been a leader in static analysis, and PC-lint has contributed to many technical advances during this time.

• The main task of CodeSonar is to search for serious defects in large code bases, whereas the tasks of the tools of the first generations are much more modest. They are primarily intended to look at inconsistencies in code for nonessential coding standards and provide more strict type control.

Here we see another attempt to mix PC-lint with those very “tools of the first generation”, as well as the arrogant statement that the tasks of PC-lint are “much more modest”. The primary task of PC-lint is to search for software errors in both small and large projects, including “real” errors like buffer overflow, overrun, logical errors, and undefined behavior. We also aim to satisfy the wide variety of requests from our customers, whose needs exceed the capabilities of many competitors. Such queries include support for various MISRA standards, strict type control, support for user semantics, and so on. Understanding that no single instrument can in principle effectively find all types of errors, we set ourselves higher goals. PC-lint seeks not only to find serious defects, but to identify programming techniques leading to their appearance. It is wrong to believe that the wide range of PC-lint capabilities means the inability to detect complex errors.

• Although first-generation tools claim to be able to find some serious errors, in reality they can only detect the most trivial problems.

Again, this statement may be true with respect to the original lint, but not with regard to PC-lint, and, implying such continuity, the authors of the report behave hypocritically and deceive readers.

Further, the authors cite several examples of errors related to the transfer of data between functions, and claim that PC-lint cannot find any of them (although in fact it finds almost all of these errors, as well as some not mentioned in the report) , and state the following:

• All the examples reviewed are extremely simple. The real code, unfortunately, is much more complicated: you have to take into account the many units of compilation, levels of abstraction, as well as extremely entangled relationships between variables due to the use of pseudonyms. And if the tools of the first generations with a shallow level of analysis are unable to find simple defects in short examples, then they are completely unsuitable for finding the whole variety of serious errors in real applications. CodeSonar, on the contrary, uses a number of advanced methods for modeling the structure of programs and therefore is able to find real errors.

This statement begins with a false premise that PC-lint during the tests could not detect the errors considered in the report, after which the authors list several unique “tricks” used by their tool, although in reality PC-lint uses the same mechanisms, some of they were implemented before the emergence of CodeSonar, contrary to the statement that they could not be found "in any primitive analyzer."

Speaking about the user interface, the authors state the following:

• The user interfaces of the compared products vary greatly. First-generation analyzers, such as PC-lint and Cppcheck, were originally intended to be run from the command line like compilers. Accordingly, the report with the results of the analysis is displayed as text. Although there are solutions for integrating these tools with a user interface that provides greater opportunities, such solutions are less effective than the interface originally developed for the analyzer and rigidly associated with it. Examples of this can be seen below.

First of all, it should be noted that PC-lint has very flexible output format settings and supports text, HTML and XML formats by default. As for graphical interfaces, along with the PC-lint analyzer itself, users receive the necessary tools for its integration with existing applications. So, we provide configurations for many popular development environments; In addition, there are a number of third-party organizations that develop serious, complete solutions for integration with development environments such as Visual Studio and Eclipse.

The report ends with the following output:

• The technology used in first-generation static analyzers, such as PC-lint and Cppcheck, has not changed much in the past 30 years. For this reason, they are unable to find serious program errors. Developers who continue to use these tools are depriving themselves of the opportunity to enjoy the fruits of decades of static analysis.

As in almost the rest of the text of the report, none of the three statements of the quoted passage contains any truth. This is an absolute fiction, based on the prior knowingly false theses.

Distortion of facts in examples

Note

The examples presented in the report are often incomplete, which is probably due to their demonstrative nature, and therefore do not allow an objective comparison of the products or the confirmation of Grammartech statements. For this reason, the authors of the report “corrected” most of the examples used so that they looked like complete, self-sufficient code fragments (as, for example, on our Online Demo demo page). To do this, they put blocks of code inside functions, declare initially undefined types and functions, correct typos, and so on. In all cases, these changes do not affect the semantics of the code or the ability of PC-lint to detect the required error, but only facilitate the task of reproducing the same results with other tools.

Buffer overflow (static)

void test_buffer_overrun(int p[]) { p[4] = 1729; } void test_driver(void) { int test[4]; test_buffer_overrun(test); } 

In relation to this fragment, the report states the following:

PC-lint cannot diagnose this error, because it does not know how to track the path of data through the procedure boundaries.

This is the most common lie. The mechanism for tracking data as they move between functions was implemented in PC-lint fifteen years ago, whereas its presence is denied in the report. So that the user can find a balance between the needs of the project and the available hardware resources, PC-lint uses a “multipass” analysis model that allows you to set the desired search depth for this type of error. By default, the search depth is 1, while for diagnosing most problems related to the relationships between functions, a search depth of 2 is required. Detailed information on the operation of this mechanism can be found in the PC-lint user guide, as well as on our official website and demo page online demo. If you run PC-lint with the key -passes = 2 (in the examples on the Online Demo it is registered automatically), the following results will be obtained:

 During Specific Walk: line 7: test_buffer_overrun([4]) #1 2 Warning 415: Likely access of out-of-bounds pointer (1 beyond end of data) by operator '[' [Reference: file ipa2.c: lines 2, 7] 2 Info 831: Reference cited in prior message 7 Info 831: Reference cited in prior message 

The message N415 warns about the required overflow, at the same time indicating how far beyond the limits of the array it arose, as well as the execution of which code sections led to an error. The last two messages (N831) are optional and are used to display warning text in a standardized format that can be recognized by development environments and other applications. In the following examples, N831 messages are disabled (via the -e831 parameter) in order to save space, since the same information is already included in the main message.

Buffer overflow (dynamic)

 typedef unsigned long size_t; void* malloc(size_t); void test_buffer_overrun(int p[]) { p[4] = 1729; } void test_driver(void) { int *p = malloc(4); test_buffer_overrun(p); } 

As in the previous case, this example incorrectly states that PC-lint is unable to find the error:

PC-lint cannot detect such errors for the same reason that it cannot detect overflows of statically allocated buffers as in the previous example.

Checking this code with PC-lint with the -passes = 2 parameter gives the following result:

 During Specific Walk: line 10: test_buffer_overrun([1]? | 0?) #1 5 Warning 662: Possible creation of out-of-bounds pointer (4 beyond end of data) by operator '[' [Reference: file ipa3.c: lines 5, 9, 10] During Specific Walk: line 10: test_buffer_overrun([1]? | 0?) #1 5 Warning 613: Possible use of null pointer 'p' in left argument to operator '[' [Reference: file ipa3.c: lines 9, 10] During Specific Walk: line 10: test_buffer_overrun([1]? | 0?) #1 5 Warning 661: Possible access of out-of-bounds pointer (4 beyond end of data) by operator '[' [Reference: file ipa3.c: lines 5, 9, 10] 

PC-lint finds both the place where the pointer is created, which points beyond the bounds of the buffer, and the place where it is used, and also checks this pointer to null (the malloc function may return null, and in the example this possibility is not checked).

A detailed description of the test_buffer_overrun ([1]? | 0?) # 1 call, which precedes the warning text, shows the way the code is executed before the message appears. In this case, we consider a call to the test_buffer_overrun function, in which a pointer is passed that points either to an array of one element ([1] ?, for example, from one value of type int ), or it is a zero pointer ( 0? ). A question mark means that it is not known which of these two options actually occurs, since the value has not been checked for null. Thus, PC-lint does not just diagnose a problem - it explains exactly how it came to a particular conclusion.

Null pointer dereferencing

 #define NULL (void *)0 void test_deref(int *p) { *p = 55; } void test_driver(void) { int *pi1 = NULL; test_deref(pi1); } 

Again, the report’s authors state that the given example of PC-lint is beyond its strengths:

This is perhaps the simplest example of dereferencing a null pointer using two procedures. Only CodeSonar can find such errors.

And again this statement can be refuted by turning on the -passes = 2 parameter:

 During Specific Walk: File ipa4.c line 9: test_deref(0) #1 4 Warning 413: Likely use of null pointer 'p' in argument to operator 'unary *' [Reference: file ipa4.c: lines 8, 9] 

PC-lint will issue a null pointer dereference warning and explain how and why it occurs.

Memory leak

 typedef unsigned long size_t; void *malloc(size_t); void free(void *); void test_free(int *p, int x) { if (p && x < 10) free(p); } void test_driver(void) { int *pi1 = malloc(20); test_free(pi1, 20); } 

For this example, the report states the following:

In this example, the buffer is allocated in one procedure, and released in another - but only when a certain condition is met. This error can only be found using CodeSonar.

However, by running PC-lint with the -passes = 2 parameter, we will see that this is not the case:

 During Specific Walk: line 11: test_free([5]? | 0?, 20) #1 8 Warning 429: Custodial pointer 'p' (line 5) has not been freed or returned 

PC-lint was able to detect this error as well, and also provide a detailed summary of the call that provoked the warning.

Reasonable comments

Each tool has strengths and weaknesses, and if you know about the shortcomings of a particular tool, it is not difficult to come up with artificial examples to emphasize them. In the report you can find some very carefully designed code examples that exploit the real, albeit well-known, disadvantages of PC-lint. Most of these examples reveal limitations in the ability of PC-lint to track pointer-related data. The data tracking system was improved in PC-lint Plus (the next step in the development of PC-lint; currently the application is in beta testing), and these limitations have been removed. The examples below are not diagnosed by PC-lint, however, we will present the results of their analysis using PC-lint Plus to show that we are continuously working to improve our product.

Uninitialized variables

 int foo() { int iret; int *p = &iret; return iret; } 

PC-Lint Plus Message:

 4 warning 530: likely using an uninitialized value return iret; ^ 2 supplemental 891: allocated here int iret; ^ 

Appeal to freed memory

 typedef unsigned long size_t; void *malloc(size_t); void free(void *); void foo() { char *p = (char *)malloc(10); char *q = p; if (p) { p[0] = 'X'; free(p); q[0] = 'Y'; } } 

PC-lint Plus message:

 11 warning 449: memory was likely previously deallocated q[0] = 'Y'; ^ 10 supplemental 891: deallocated here free(p); ^ 6 supplemental 891: allocated here char *p = (char *)malloc(10); ^ 

Dual Memory Release

 typedef unsigned long size_t; void *malloc(size_t); void free(void *); void test_double_free(int *p) { if (p) free(p); } void test_driver(void) { int *pi1 = (int *)malloc(sizeof(int)); if (pi1) test_double_free(pi1); if (pi1) free(pi1); } 

PC-lint Plus message:

 15 warning 2432: memory was potentially deallocated free(pi1); ^ 15 supplemental 894: during specific walk free([4]@0/1) free(pi1); ^ 7 supplemental 891: deallocated here free(p); ^ 11 supplemental 891: allocated here int *pi1 = (int *)malloc(sizeof(int)); ^ 

Buffer overflow

 void foo() { char buffer[10]; char *pc; pc = buffer; for (int i = 0; i <= 10; i++) *pc++ = 'X'; } 

This example demonstrates a buffer overflow in a for loop. The models used in PC-lint and PC-lint Plus to track the state of values ​​(so far) do not take into account the relationship between i and pc in this code. At the moment, this is an objective disadvantage of PC-lint. We could easily come up with a similar example to highlight the weaknesses of CodeSonar, but this would not have given us anything. As mentioned above, we recognize that each tool has its own advantages and disadvantages and that each tool can diagnose errors that are invisible to others. Instead of pointing out the shortcomings of other analyzers, we prefer to work on the merits of PC-lint and continuously improve our product to meet the needs of users.

Summary

In the table below, we have collected the Grammatech key false statements regarding PC-lint and the facts that disprove them on our part.

False statement	In fact
PC-lint is a primitive tool from the original lint family in Unix.	PC-lint is an advanced static analyzer that continuously and independently of other tools has been developing and improving over the past 30 years, offering innovative, award-winning analysis tools. One of them is a system for tracking data during its transfer between functions and modules.
PC-lint can detect only the most obvious errors.	PC-lint uses a variety of advanced technologies that allow it to detect complex errors, including errors from the examples in the Grammartech report.
PC-lint is not intended to search for serious errors in large projects.	PC-lint is successfully used on projects of any size, from hundreds of lines of code to millions.
PC-lint only supports text output format and is not designed for use with continuous integration tools.	PC-lint supports almost any report format, including plain text, HTML and XML, and, as our clients experience shows, can be used in a variety of ways in conjunction with other applications, including continuous integration tools such as Hudson and Jenkins.
PC-lint does not show the way of execution of the code that triggered the warning when searching for complex errors.	PC-lint provides detailed information, where possible, including a sequence of calls and values ​​leading to a conclusion.
PC-lint is for developers only.	PC-lint, in accordance with its purpose, is used by software developers, technical control experts, testers and forensic experts.
PC-lint can only recognize problems within a file.	From the very beginning of its existence, PC-lint is able to analyze programs entirely, including the relations between modules. This feature, among others, distinguishes it from other tools.
PC-lint is unable to find all possible software errors.	Given that no static analyzer is able to diagnose all errors, PC-lint, nevertheless, has a long track record of detectable defects, including many complex, real-life errors, and its capabilities continue to be improved.

Conclusion

Gimpel Software welcomes sincere reviews and constructive criticism from both our users and competitors, but the deliberately false statements made by the authors of the reviewed report are neither sincere nor constructive and do not serve the interests of programmers or the static analysis industry. The Grammartech policy, based on false and disparaging statements about competing products, puts this company at a disadvantage, not its competitors.

As you can conclude from the report, the main differences between PC-lint and CodeSonar are as follows:

• CodeSonar is equipped with a tightly integrated graphical interface, while PC-lint provides the user with the flexibility to integrate the analyzer with modern tools, such as Visual Studio, Eclipse and others.
• The scope of CodeSonar is narrower than that of PC-lint, therefore, its functionality is less than that of our product.
• CodeSonar has a number of functions not yet supported by PC-lint. In particular, the report mentions the analysis of metrics and the taint analysis - we are developing both of these mechanisms for a future version of PC-lint Plus. Of course, many of the functionality available in PC-lint is missing in CodeSonar.

Translator's Note

Here is a good example why we don’t like to write about any comparisons of analyzers. Too this is a complex task, where it is necessary to know different tools equally well, not to be subjective, and so on. I believe that at least some plausible assessment can only be obtained by comparing the results of testing a large number of projects with different analyzers. But this is a sooo big task with a lot of nuances. All other assessments are only a subjective opinion, which, as we see here, can easily turn into a conflict situation.