John Carmack on static code analysis
“The most important thing I did as a programmer in recent years was to aggressively use static code analysis. - John Carmack writes in an article published on AltDevBlogADay. “Hundreds of serious bugs are not so important, which were prevented by how much a change in mentality and my attitude to software reliability and code quality.”
John Carmack talks about the various tools that he tried to use during his career and the conclusions he came to in the end.
From the very beginning, Carmack stipulates that the quality of the code is not the most important thing in software development, and “the recognition of this fact is not some kind of moral defeat. Value is what you are trying to create, and quality is just one aspect, along with cost, functionality, and other factors. There were a lot of extremely successful and respected games full of bugs and constantly departing; so the use of space shuttle programming style in game development would be idiocy. However, quality still matters. ”
The developer of the Quake engine says that he always tried to write good code, since one of his inner motivations was that as a master of his craft he had to constantly improve. Carmack read “a bunch of books with boring titles” like “Policies, standards and quality plans”, and working with Armadillo Aerospace allowed him to get acquainted with a completely different software development world, where security is a critical factor.
')
Carmack's introduction to static code analysis tools began more than ten years ago. Then, during the development of Quake 3, he bought the PC-Lint program and tried it out. However, the launch from the command line and the need to wade through a bunch of generated comments did not at all please Carmack. “I gave up and quite quickly abandoned PC-Lint,” he writes.
But in the future, I still had to revise my opinion, because the code base grew by an order of magnitude, and the code was transferred from C to C ++, so that new errors could appear during the migration process. A few years ago, Carmack decided to see how static code analysis tools have changed over the years.
First of all, he signed up for the Coverity demo, this is serious software, whose license cost depends on the number of lines of your code, so id Software has reached the five-digit number. The program showed its best and immediately found a hundred bugs. Carmack immediately noted that in ten years, much has changed, the instruments have become much more efficient in terms of the signal-to-noise ratio. All is good, but the high price still scared away.
After that, we tested the Microsoft / analyze library, which is now included as part of the 360 SDK. Previously, this library was part of a top-end, fabulously expensive version of Visual Studio, but is now available to every 360 owner at no extra charge. “I regard it as such that for Microsoft the quality of Xbox games is more important than the quality of programs for Windows,” joins John Carmack.
As it turned out, the utility from Microsoft showed itself even better than Coverity in the number of errors found, although it also gave a large number of false positives. John Carmack worked for several months, gradually correcting the found bugs: first his code, then another system code, then the game part. The effect was colossal, there were very important mistakes that were simply forgotten at the time, as well as new ones. Working with the / analyze option is simply a must for everyone, it effectively protects against the appearance of bugs in the development process.
Since the tool from Microsoft was used only for Xbox 360 code, the question of what to use for PC and PS3 was still open. Next we tried the PVS-Studio program, which integrates well with Visual Studio and has a convenient demo mode. Compared to / analyze, PVS-Studio proved to be an extremely slow tool, but it found several additional important errors, including logical ones, even in code that was completely clean from the point of view / analyze. On the PVS-Studio website there are a number of good articles with real examples of exactly what types of errors it can find.
John Carmack also gave a second chance to the good old PC-Lint , in conjunction with Visual Lint for integration with the IDE. According to the good old yuniksovoy tradition, this tool can be configured for almost any task, but in fact is very friendly. And again, even in the code that was clean after checking and / analyze, and PVS-Studio, there were new errors. However, it is rather difficult to work with PC-Lint in the sense of finding the most effective settings for the level of alerts.
In the end, John Carmack made several conclusions, which he advises others to understand. First of all, everyone must admit that there are many errors in his code. This is a bitter pill that every programmer needs to swallow. Secondly, the code revision must be automated. It is nice to see the erroneous responses of automatic systems on your code, but for every error of the automatic system there are about a dozen human errors. Tips to “write the best code”, work in pairs and so on, do not work here, especially when dozens of programmers are in conditions of lack of time.
Carmack also notes that after each update, PVS-Studio finds new errors. That is, errors in the code you can never eliminate to the end. In a large project, they will always be, as statistical deviations in the properties of the physical material. You can only try to minimize their negative effect.
The biggest problem in C / C ++, at least in our code, says Carmack, is NULL pointers, followed by printf string format errors in second place. A large number of serious errors occur in the process of modifying the old code.
And finally, John Carmack quotes Twitter from Dave Revell (Dave Revell): "The more I use static code analysis, the more surprised that computers generally load."
John Carmack talks about the various tools that he tried to use during his career and the conclusions he came to in the end.
From the very beginning, Carmack stipulates that the quality of the code is not the most important thing in software development, and “the recognition of this fact is not some kind of moral defeat. Value is what you are trying to create, and quality is just one aspect, along with cost, functionality, and other factors. There were a lot of extremely successful and respected games full of bugs and constantly departing; so the use of space shuttle programming style in game development would be idiocy. However, quality still matters. ”
The developer of the Quake engine says that he always tried to write good code, since one of his inner motivations was that as a master of his craft he had to constantly improve. Carmack read “a bunch of books with boring titles” like “Policies, standards and quality plans”, and working with Armadillo Aerospace allowed him to get acquainted with a completely different software development world, where security is a critical factor.
')
Carmack's introduction to static code analysis tools began more than ten years ago. Then, during the development of Quake 3, he bought the PC-Lint program and tried it out. However, the launch from the command line and the need to wade through a bunch of generated comments did not at all please Carmack. “I gave up and quite quickly abandoned PC-Lint,” he writes.
But in the future, I still had to revise my opinion, because the code base grew by an order of magnitude, and the code was transferred from C to C ++, so that new errors could appear during the migration process. A few years ago, Carmack decided to see how static code analysis tools have changed over the years.
First of all, he signed up for the Coverity demo, this is serious software, whose license cost depends on the number of lines of your code, so id Software has reached the five-digit number. The program showed its best and immediately found a hundred bugs. Carmack immediately noted that in ten years, much has changed, the instruments have become much more efficient in terms of the signal-to-noise ratio. All is good, but the high price still scared away.
After that, we tested the Microsoft / analyze library, which is now included as part of the 360 SDK. Previously, this library was part of a top-end, fabulously expensive version of Visual Studio, but is now available to every 360 owner at no extra charge. “I regard it as such that for Microsoft the quality of Xbox games is more important than the quality of programs for Windows,” joins John Carmack.
As it turned out, the utility from Microsoft showed itself even better than Coverity in the number of errors found, although it also gave a large number of false positives. John Carmack worked for several months, gradually correcting the found bugs: first his code, then another system code, then the game part. The effect was colossal, there were very important mistakes that were simply forgotten at the time, as well as new ones. Working with the / analyze option is simply a must for everyone, it effectively protects against the appearance of bugs in the development process.
Since the tool from Microsoft was used only for Xbox 360 code, the question of what to use for PC and PS3 was still open. Next we tried the PVS-Studio program, which integrates well with Visual Studio and has a convenient demo mode. Compared to / analyze, PVS-Studio proved to be an extremely slow tool, but it found several additional important errors, including logical ones, even in code that was completely clean from the point of view / analyze. On the PVS-Studio website there are a number of good articles with real examples of exactly what types of errors it can find.
John Carmack also gave a second chance to the good old PC-Lint , in conjunction with Visual Lint for integration with the IDE. According to the good old yuniksovoy tradition, this tool can be configured for almost any task, but in fact is very friendly. And again, even in the code that was clean after checking and / analyze, and PVS-Studio, there were new errors. However, it is rather difficult to work with PC-Lint in the sense of finding the most effective settings for the level of alerts.
In the end, John Carmack made several conclusions, which he advises others to understand. First of all, everyone must admit that there are many errors in his code. This is a bitter pill that every programmer needs to swallow. Secondly, the code revision must be automated. It is nice to see the erroneous responses of automatic systems on your code, but for every error of the automatic system there are about a dozen human errors. Tips to “write the best code”, work in pairs and so on, do not work here, especially when dozens of programmers are in conditions of lack of time.
Carmack also notes that after each update, PVS-Studio finds new errors. That is, errors in the code you can never eliminate to the end. In a large project, they will always be, as statistical deviations in the properties of the physical material. You can only try to minimize their negative effect.
The biggest problem in C / C ++, at least in our code, says Carmack, is NULL pointers, followed by printf string format errors in second place. A large number of serious errors occur in the process of modifying the old code.
And finally, John Carmack quotes Twitter from Dave Revell (Dave Revell): "The more I use static code analysis, the more surprised that computers generally load."
Source: https://habr.com/ru/post/135234/
All Articles