Why I do not like synthetic tests

I don’t like it when someone tries to use hand-crafted code samples to evaluate the capabilities of a static code analyzer. Now on a specific example, I will demonstrate why I have a negative attitude to synthetic tests.

Not so long ago, Bill Torpey wrote a note in his blog " Even Mo 'Static ", where he told how, in his opinion, the Cppcheck and PVS-Studio tools showed themselves when analyzing the itc-benchmarks project. The itc-benchmarks project is static analysis benchmarks from Toyota ITC.

I didn’t like that after reading the article it seems that the Cppcheck and PVS-Studio analyzers are approximately equal in their capabilities. It follows from the article that one analyzer shows itself better in one, the second in another, but on the whole their diagnostic capabilities are similar.
')
I think it is not. My opinion - our PVS-Studio analyzer is several times more powerful than Cppcheck. And in general, this is not an "opinion", I know that!


However, since it’s not visible from the outside that PVS-Studio is 10 times better than Cppcheck, then you should try to understand the reason. I decided to look at this very itc-benchmarks and figure out why PVS-Studio showed itself on this test base not in the best way.

The further I understood, the more irritated I felt. And one example completely put me off balance, and I will tell about it a little lower. My conclusions are: I have no complaints about Bill Torpey. He wrote a good, honest article. Thank you, Bill. But I have a claim to the Toyota ITC. My personal opinion: their test base is crap. This is, of course, a loud statement, but I believe that I have sufficient qualifications and experience to talk about static code analyzers and how to evaluate them. In my opinion, itc-benchmarks cannot be used to adequately assess the capabilities of a particular analyzer.

But the actual test, which finally brought me out of balance.

This is a null pointer dereference test :

void null_pointer_001 ()
{
  int *p = NULL;
  *p = 1; /*Tool should detect this line as error*/
          /*ERROR:NULL pointer dereference*/
}

Cppcheck , :

Null pointer dereference: p

PVS-Studio , V522.

, PVS-Studio Cppcheck? , !

PVS-Studio , .

, , . . . , , :

void GpuChildThread::OnCrash() {
  LOG(INFO) << "GPU: Simulating GPU crash";
  // Good bye, cruel world.
  volatile int* it_s_the_end_of_the_world_as_we_know_it = NULL;
  *it_s_the_end_of_the_world_as_we_know_it = 0xdead;
}

PVS-Studio V522 , . , null_pointer_001 . , . , « » .

, V522 A6. null_pointer_001. A6:

, :

• error
• default
• crash
• null
• test
• violation
• throw
• exception

, 0 .

. -, «null». -, . . , .

- !

itc-benchmarks . , , :

void null_pointer_006 ()
{
  int *p;
  p = (int *)(intptr_t)rand();
  *p = 1; /*Tool should detect this line as error*/
          /*ERROR:NULL pointer dereference*/
}

rand 0, NULL. PVS-Studio , rand .

, rand. , , . , .

. , - , . , PVS-Studio itc-benchmarks, , - .

, , rand . , . . .

, rand 0, 1400 . . - , .

. , , , , V501. , itc-benchmarks , , «if (a.x == a.x)». !

, itc-benchmarks . , . , , , . , , :

void overrun_st_014 ()
{
  int buf[5];
  int index;
  index = rand();
  buf[index] = 1; /*Tool should detect this line as error*/
                  /*ERROR: buffer overrun */
  sink = buf[idx];
}

, .

, :

return (!strcmp (a->v.val_vms_delta.lbl1,
                 b->v.val_vms_delta.lbl1)
        && !strcmp (a->v.val_vms_delta.lbl1,
                    b->v.val_vms_delta.lbl1));

PVS-Studio GCC. .

, rand , .

, , , . . . -, .

PVS-Studio.

:

1. PVS-Studio.
2. , PVS-Studio .
3. . – , .

, : Andrey Karpov. Why I Dislike Synthetic Tests