Self-deception of simplicity

The article deals with the problem of exceeding the deadlines for creating programs as a result of self-deception: as if programming is simple and even simpler. The creators of new technologies in the field of software development assure that their technology will help solve a huge number of problems and that software development is now becoming easier and faster than ever. But the practice is relentless. Again and again the deadlines for the completion of the next program project are being broken.

Introduction

Disrupting the timing of software development is a common occurrence in the practice of creating software products. This phenomenon is harmful for all participants in the process, ranging from ordinary developers who are forced to work in a rush job, and ending with management and customers who are forced to put up with lags and additional costs.

There are many reasons for this phenomenon, but in this article I want to express the opinion that the main component is the common belief that software development is simple. Immediately emphasize an interesting point. In general, everyone understands that creating programs is a laborious process that requires qualifications and time, but as soon as it comes to discussion or presentations, everyone, pursuing his own interests, begins to say that all the difficulties are in the past and now using new technology / programming language / methodology, everything will become easy and simple. This is repeated over and over again for many years. In general, there is no improvement, and the deadlines are still broken. It is the causes of these phenomena and their consequences, we will discuss in this article.
')

Why I thought about these questions

I will proceed not from general reasoning, but from personal experience. I am engaged in creating a tool for PVS-Studio programmers. This is a static C / C ++ code analyzer for testing programs written for 64-bit systems or created using OpenMP technology. In such programs, a number of error patterns arise, which, nevertheless, it is sufficient to simply diagnose by analyzing the source code. But that's not the point. The fact is that by participating in the promotion of these tools, I was faced with a lot of resistance caused by the fact that the use of 64-bit technologies and OpenMP is presented as an extremely simple thing. As a result of this confidence, it is necessary to overcome the enormous resistance of people who believe that there are no problems and there can not be. I turned out to be the enemy of their peaceful world. A similar picture, of course, is observed with other technologies, but in the article I will talk about OpenMP technology and 64-bit programs as the most familiar area for me.

The belief that everything is simple is not surprising. Around describe how easily you can make your program 64-bit and immediately get a performance boost and a huge amount of available memory. Describes how, using OpenMP, you can easily turn your code into parallel. Around articles containing abstracts in the spirit of “just recompile the application”, “simple arrangement of OpenMP directives”. And on this beautiful celebration of life, I go with a poster, with large headlines on the articles “20 C ++ porting code traps to the 64-bit platform”, “32 OpenMP pitfalls in C ++ programming”. At the same time, I feel like a nasty old woman who is unhappy with everything and constantly grumbles at everything. It was this strange feeling and resistance to the “evangelical simplicity” that made me think and write this text.

There was a moment when I was thinking about the idea that maybe I’m not really right. It is necessary to talk with everyone like everything is simple. And that with our tools, everything is just completely. But something is wrong. There is a contradiction. I am well aware of the state of affairs, and I am confident that there are difficulties awaiting programmers in mastering parallel technologies and 64-bit systems with a large amount of memory. There is no simplicity. This is a lie. And cheating is bad. It is harmful to me, as a person interested in promoting his tools. It is harmful to the manager who cannot correctly predict the dates. And, of course, it is harmful to the programmer, who is disappointed in new technologies and overtime. Therefore, I decided to stay in my positions and will also try to change your point of view.

Where does complexity come from?

I can still try to convince me that the creation of 64-bit programs is not so difficult that a number of programs can be rebuilt for 64-bit systems without fixes. And errors can be automatically found and corrected using different techniques. Although it will not be easy, since I have experience talking about the complexity of these processes and how easy it is to err in estimating dates several times.

But when parallel programming is also presented, this is a complete discrepancy of reality. Parallel programming is a complex and overly complex task that cannot be solved by simply placing OpenMP directives around loops or using a library. But this is how it is presented. Many developers who have not yet studied the issues of creating parallel algorithms are told that it is not necessary to think about it and the situation will fix OpenMP or some other technology. From such articles it seems simple and clear how to parallelize any program. But the popularization of what the reader himself cannot do and test beforehand creates only an illusion of understanding. Perhaps, it enriches his vocabulary and somehow expands the circle of his thinking, but at the same time it also creates an unnecessary illusion of understanding where there is no real understanding and knowledge.

This does not mean that OpenMP technology or any other is bad. They are wonderful. But you need to understand what they can and what they don’t. Do not expect to easily parallelize your existing programs. Most likely, this will require the creation of new algorithms and the transformation of data structures and mechanisms for working with them. The ease of embedding OpenMP into existing code should be understood so that this integration will result in fewer changes than the use of other technologies. So MPI technology facilitates the construction of a distributed system, compared with the implementation of the same functionality through direct work with the API and network protocols. In turn, MPI is unnecessarily complex for multi-core systems with shared memory and requires significant changes in the code. Compared to it, OpenMP is really extremely easy to integrate. But only in comparison and only in a certain class of tasks!

It is impossible to talk about the simplicity of parallel programming, despite what technologies are used. This area is complex in itself. If you want to make sure, try to quickly sit down and parallelize the array sorting algorithm that everyone wrote in the school and the institute. If that doesn't work, then look at the Batcher parallel sorting. Immediately you can imagine? You can not imagine. This example is made to show what kind of difficulties you should expect even with the familiar algorithms used in your programs. But there are a lot of other aspects.

You probably use iterators in cycles in your C ++ programs? And you know that in popular articles about OpenMP, when they show examples of parallelization for, you are forgotten to warn that iterators cannot be used in it? Indexes should be simple data types (for example, int). That is, when planning time for reworking algorithms, please note that you may also need to change the data structures and methods of working with them. And all this extra time and effort. Yes, OpenMP 3.0 has the opportunity to work with iterators. Articles before the advent of OpenMP 3.0 avoided the topic of their use. New articles are written in the style of "it became easier to write programs on OpenMP 3.0, due to the possibility of using iterators." Yes, it is good that now there is such an opportunity. But now imagine the number of nuances and difficulties that exist, but at the same time are practically not described anywhere and that you will have to face on your own.

If everything is difficult, then why say that simple?

There are no villains who want to deceive you and convince you that everything is simple. But there are various interests that lead to mutual deception. Anyone who promotes 64-bit hardware platforms and operating systems is naturally interested in convincing that transferring your applications to them is extremely simple. It's clear. The transition to 64-bit systems has already dragged out and, if we say that there are difficulties, this can extend the transition for another several years.

Manufacturers of parallel systems advertise various software technologies and libraries, convincing that everything will become parallel almost by itself. Those who create these libraries / technologies try to convince programmers that their library / technology is the best, as it is the simplest. And here comes the sad confusion. The fact that the library is simple and convenient (for example, in my opinion, OpenMP is simple and convenient) does not mean that the creation of parallel programs is also easy!

Programmers, having read a lot of articles about how all is well in the 64-bit and parallel worlds, but without having yet practice, deceive the leadership and themselves about the timing. Leadership cheats customers. And the leadership itself likes to consider everything simple, as it seems to them that this allows them to save on developers. As a result, there are no guilty ones, but everyone deceives each other.

findings

Already in the process of writing this text, I found a wonderful article by Dijkstra “Two Views on Programming” [1], which largely overlap with the meaning of this article. I will quote a small excerpt that sums up all of the above very well: “Basically, it’s not the fault of computer manufacturers who want to do business as if they were selling the simplest products; and not the fault of software project managers who prefer to view the activities of programmers as a simple and predictable process; and not the fault of educational institutions that would like to prepare students for achieving guaranteed success. This is a consequence of the comfortable illusion that the Man is only a complex automaton, an illusion that, like a drug, brings its victims an apparent release from the burden of responsibility. The recognition of programming as a serious challenge to the intellect would return the full weight of this burden back to their shoulders. ”

But the weight of the burden should not be the reason why you should keep your eyes closed. From this lose everything. It is necessary to accept the complexity as it is.

The conclusion from all this should be a more critical look at the possibilities of the latest technologies. New tools and techniques are useful, but they could never and probably cannot replace the software development process itself, consisting of many stages and elements. First of all, it is necessary to study and improve the development process, and only then choose new convenient technologies for the project implementation.

Irreplaceable helpers in getting out of the circular “deception of simplicity” are the remarkable books by Steve McConnell “How much does a software project cost” [2] and “Perfect code” [3]. And, of course, the cult book by Frederick Brooks "The mythical man-month, or how software systems are created" [4].

Bibliographic list

1. Edsger Dijkstra. Two views on programming. URL. http://www.inr.ac.ru/~info21/pdf/dijkstra.pdf
2. McConnell S. How much does a software project cost? - M .: “Russian Edition”, St. Petersburg: Peter, 2007. - 297 pp., Ill.
3. McConnell S. Perfect code. Master Class / Trans. from English. - M .: Publishing house "Russian edition", SPb .: Peter, 2007.- 896 p .: ill.
4. Frederick Brooks. Mythical man-month or how to create software systems. - Per. from English - SPb .: Symbol Plus, 1999, - 304 pp., Ill.