How much is one error in our programs, or what errors happen in PVS-Studio and CppCat

In the list of questions that always ask us after reading our articles there is a question: “Do you check the code of your analyzer by yourself?”. And the answer is given in the vein, that we always check the code while writing using the incremental analysis mode (checking the files immediately after compilation). And also run night checks for the whole code. And in this connection we, unfortunately (irony), can never write an article about errors in our software products.



However, today it is an article about one of our mistakes and about how much it cost us. Intrigued?





')

My story today will be in faces, so first the “credits”:

• Michael, user CppCat;
• Evgeny (I), the author of the article, is related to PVS-Studio and CppCat;
• Andrew, my colleague, sometimes writes different articles;
• Pavel, my colleague, knows everything about integration into Visual Studio;

Once on Friday, February 21, we published an article " A side effect: we check Firebird with PVS-Studio ", in the comments to which Mikhail appeared with the following comment :



CppCat - what you need! You would make a separate post about it. I will ask for guidance, tea is cheap :)



Smiling, my colleague Andrei answered him that already three articles were about CppCat. True colleague immediately received a "scolding" for the fact that no one knows about our new product. It was at 16:16 Moscow time ...



Already at 4:21 pm in my CppCat-mail there was a letter from Michael with the Wishlist. Michael wanted the CppCat error messages to be displayed not in a separate window, as now, but in the standard Visual Studio error window. From my point of view as the “custodian of the product” CppCat, this wish did not seem to be expedient, and I tried to explain in the mail that we had a specially made separate window so as not to confuse our messages and Visual Studio messages. That it is very convenient, as it allows us to make the window specialized, etc. But Mikhail "did not let up," giving a simple argument:



- And how can I see then after recompiling the error file? If only they would immediately be thrown out the window ...



But for this very purpose, we have a “killer feature” incremental analysis! After recompilation of files, the code analyzer is automatically launched. And only if there were real errors, a notification is issued in the tray and the title of the analyzer window changes. A phrase appears in the window stating that new errors have been found. Incremental analysis mode can be turned on / off in the menu.



“Well, yes,” answered Mikhail, “I have an“ Incremental Analysis ”checkbox. But how do I know if there are errors?



After five letters, including screenshots, it became clear that although everything is set up correctly, no notification is issued in the tray, nor is the window text changed. And the time was six o'clock on Friday evening ...



With the thought: “I haven’t taken checkers for a long time (from the day before yesterday)!”, I asked for Michael to the computer via remote access from AMMYY Admin. Stumbling into my program, I realized that there really are no notifications about new errors, although the program seems to be working. The time was nearing six in the evening, and I went to my colleague Pavel, who is responsible for the integration in Visual Studio. Showed him what I was doing on the user's computer and that the notification was not working.



Thirty seconds later, Paul said:



- Eugene ... Well, you forgot that with the “Rebuild All” / “Rebuild All” command, we don't specifically start the incremental analysis. This is done consciously and everything is correct. And in general, you need to look at your own product more often. And already forgot everything. If only to answer the letters in the mail, yes write articles about business ...



Reddening, I went to check ... However, there was no notification either when reassembling one file or when reassembling several files. I tried on another project - the result is the same. Does not work. And the very important function that is the killer feature of our tool does not work. Let me remind you that I tried all this on a remote user machine. I went to Paul again, but it was almost six. And the user wanted to go home, and Paul ... Perhaps in this regard, we together could not understand why it does not work. Deciding on Monday to issue a version with debug logs, we went home.



All weekend I thought about this issue. Is our product to blame? If so, when could this problem appear in the code? And how many people DIDN'T BUY our product, as a very important thing did not work as it should.



On this sad note came Monday. We gave the user a special version with debug logs. He sent them to us. The logs were not clear, unfortunately. We made more detailed logs, described the instructions for reproducing as detailed as possible ... It did not help. The logs looked as if the code was not working, although it could not work.



For Monday and Tuesday, we gave Michael several versions with the most detailed debug logs. But to understand what is happening could not. Notifications of new errors simply did not exist. Him. And we had, always.



In the middle of Tuesday, Paul called me:



- Yevgeny, you won’t believe ... But I don’t see a notice about errors found in CppCat!



Do you think the debug logs helped us? No, Pavel RANDOMLY clicked on the auto-hide button in the CppCat window inside Visual Studio. And our window was automatically hidden when you move the mouse away from it. Since none of the developers use the auto-hide button with us, we could not have imagined that in this mode our notification doesn’t work!



That is, if auto-hide is turned off (normal mode), there are notifications of new errors, and if auto-hide is turned on, there are no notifications of new errors.



I asked the user Michael to check whether the matter is in auto-hide. He confirmed that in him.



After that, it was already a matter of technique to correct the error. It turned out that the plugin code did not correctly handle a particular window style. For programmers, I asked Paul to describe the problem and its solution, but I don’t provide this text here, as it will be understood only by the developers of plug-ins for Visual Studio.



So, we were able to repeat the mistake on Tuesday afternoon. On Wednesday, issued a revised beta version of Michael.



While Pavel was doing the patch, I talked with Mikhail about the licensing policy of our CppCat and PVS-Studio products. Although Michael got the mistake when he became acquainted with CppCat, upon further communication, it turned out that PVS-Studio could be useful for his team.



Let me remind you for those readers who are not familiar with what we are doing. We develop, promote and sell two of our own software products CppCat for individual developers for $ 250 and PVS-Studio, whose minimum license for a team of up to 9 people costs a little more than € 5,000.



So after discussing the licensing issues, Mikhail apparently sent a letter to the management and on Tuesday evening I discussed with his boss licensing issues of PVS-Studio. We talked about everything pretty quickly, they came to the conclusion that they didn’t need CppCat, but PVS-Studio.



But what about that mistake? On Wednesday we gave Mikhail a revised beta version, he checked that everything works. On Wednesday, I sent my boss Mikhail a bill to pay for PVS-Studio. On Thursday, I saw a license fee on our account. And I decided that it would be nice to describe this story, all of a sudden it will seem interesting to someone.



So fixing one mistake in a $ 250 product gave us the sale of another product of ours for completely different money. And all this in one week, because today, on Friday, February 28, you are already reading an article about this story!



If you ask why I wrote all this, my answer is very simple. I am proud of my team that we can quickly correct our mistakes and that our customers appreciate it.



PS Many thanks to our user Michael, who helped us diagnose the error, without him this story would not exist. Our client programmers are the best users in the world!



UPDATE1: For some reason, in the comments I write about the fact that it was necessary to give a discount. Of course, a discount was given, what other options might there be? :-)



UPDATE2: Michael was noted in the comments below. Give him a plus in karma, for what helps developers catching bugs :-).

Unfortunately, we no longer develop or support the CppCat project. You can read about the reasons here .