Generation Lost in the Bazaar
“Quality comes only when someone is personally responsible . ”
- Frederick F. Brooks
Hi, Habr!
')
I bring to your attention a free translation of Paul A-Henning Kamp's e- essay " A Generation Lost in the Bazaar ", telling us about the sad fate of a generation of IT professionals who grew up during the dotcom boom , as well as fundamental UNIX issues that directly affect software quality and portability . Everything in order.
In my article, the author uses the cathedral and bazaar metaphor described in Erik Raymond’s essay “ Cathedral and Bazaar ”, I find it appropriate to provide an excerpt from the text of a recognized translation of the essay:
--- start of quotation ---
Linux is an amazing system. Who would have thought that several thousand developers scattered all over the planet and collaborating only via the Internet would be able to create a world-class operating system. I did not think so at all. By the time Linux came to my attention in early 1993, I had been involved in the development of UNIX and open source software for about ten years.
Linux turned my ideas about what I know. I thought that the main thing in developing small tools for UNIX was their quick design and evolving programming over many years. And at the same time, I believed that as the complexity of the development increases, a more centralized approach is needed. I believed that developing the most complex software (for example, operating systems or simply large tools such as Emacs) should be like building a cathedral. Such programs should be created by individualistic masters or small groups of wizards working in strict isolation, avoiding premature beta versions.
I was very surprised by the development style of Linus Torvalds - the frequent release of releases, the availability of all source texts and the tolerance for heterogeneous programs. This is quite unlike the measured construction of the cathedral, the Linux community is more like a noisy bazaar, with many different approaches and directions. The fact that an agreed stable operating system is born on this market seems like a miracle of wonders.
I was shocked that this market style works and works well. I not only participated in the development of individual projects, but also tried to understand why in the Linux world not only confusion does not arise, but on the contrary, it moves forward at a speed that the cathedral builders can only envy.
By the middle of 1996, it seemed to me that I had begun to understand. Fate gave me a great chance to test my theory. It was an open source software development project that I deliberately spent in the marketplace style. The success of this project exceeded all expectations.
--- end of quotation ---
--- Start of translation ---
Eric Raymond's book, The Cathedral and the Bazaar , published thirteen years ago, has expanded our vocabulary with new terminology. In addition, she predicted the end of the cascading software development model and the decline of the era of large software companies, thanks to the Free Software movement that had emerged among the people.
After reading the book I had something to think about, but she did not convince me. On the other hand, I have a direct relationship to the world of open source software, so even though I cannot help, it would be great if the author were right.
Another book that turned out to be in my house by the sea this summer also brings thoughts. She is clearly deeper than Raymond’s book (by the way, positively refers to him), this is Frederick P. Brooks's - “The Design of Design” . And how many times I nodded in agreement, enjoying Brooks’s writing style and his ability to submit material, the same time I was depressed by what I read.
Thirteen years ago, the dot-com era of the 90s was reached, when every schoolchild was a web programmer or everyone who had been expelled from college already owned a web start-up. I got real pleasure when I tried to teach these salads with fundamental professional skills - creating automatic deployment scripts, version control systems, and so on. Nostalgia, of course - in fact, it was not fun.
We cannot deny that the whole dot-com era was a real disaster for the IT / CS industry in general and for UNIX in particular. The quality of software products was hit.
I do not have exact information on how much the IT industry has increased in volume during the period of the dot-com. My personal assessment is, if we count the number of professions that have surfaced, that our niche has grown by two orders of magnitude, that is, by more than 10,000% .
It's easy to fall in love with computers - almost anyone can make a work program, just like almost anyone can nail one board to another, in just a few attempts. The problem is that the share of such unprofessionally nailed boards is extremely small, if we consider the market of home furniture. And to move from two boards to a decent-looking set of chairs or a built-in wardrobe requires talent, practice and education. Those same (10 000% - 100%) = 9 900% additional percent had neither experience nor education when they entered the IT industry. And before they had a chance to get it, the party was over and most were left without work at all.
I will show mercy and note that those who clung to the position were undoubtedly the most talented and most skillful of all, but even in this case, it cannot be denied that they did not take place as IT professionals because of the lack of life experience.
The market style that E. Raymond so advocated, “ Just started drinking ”, opposed to well-designed cathedrals from the pre-Dotcom period, did not disappear after the dot-com crash, unfortunately, even today UNIX is sinking under its own weight.
I updated my laptop. For 18 years now, I have a dev-version of FreeBSD on it and it takes all day to build a minimalist work environment from source, because it tries to build logic and architecture from anarchic garbage, called E. Raymond as a bazaar.
From afar, it seems that the FreeBSD ports collection is something like a map for a bazaar, so that FreeBSD users find it easier to find something. In practice, this “map” consists, at the moment, of 22,198 files with a brief description of each tent at the bazaar — a few lines about what is inside and where to read more about it. There are also 23 214 make-files telling us what to do with each software from those that we find in tents. These makefiles try to inform us about possible scenarios, various custom options and their default values. The card comes with 24,400 patch files to smooth out the curvature of the proposed handicrafts. Although, in fact, most of these patches are used to fix software portability issues.
Finally, this map helpfully tells us that if we want to install www / firefox , then first we need to deflate devel / nspr , security / nss , databases / sqlite3, and so on. As soon as we find them using the map, and then look for their dependencies, and then recursively get a list of dependencies of their dependencies, then we will have a list of purchases for 122 packages that we definitely need before we can get www / firefox .
We all love the component-oriented approach to software development. But even in the most trivial case, however, the code reuse methodology is completely alien in the market: the programs in the FreeBSD ports collection contain at least 1,342 copy-pastes of cryptographic algorithms.
If such a disregard for the reuse methodology would have been embodied as a mechanism of self-contained and independent software packages, then there would be a trade-off between code duplication and the ease of package management. But this is clearly not our case - the packets form an intricate web of unsystematic dependencies, which leads to even more duplication of code and a waste of resources.
Here's an ironic example: Sam Liffera's graphics / libtiff library is on its way to www / firefox , being one of those 122 packages in a common dependency chain. While the target program — the Firefox web browser — cannot display TIFF pictures.
. I did not try to figure out the reasons why 10 packages out of 122 require Perl and 7 require Python ; And one of them, devel / glib20 , requires both languages, I can not even imagine why.
Next on the list ... here is Peter's Principle , which sounds like "In a hierarchical system, any worker rises to the level of his incompetence," and globally it can be formulated as: "Any well-working thing or idea will be used in increasingly difficult conditions until will cause a catastrophe. " In software engineering, a special case is used - “A dying project, which is gradually becoming too complicated for it to be understood by its own developers”. You see, we need 3 different versions of make, a macro processor (m4), an assembler, and a bunch of other interesting packages. At the end of the food chain, so to speak, is the libtool tool, which is designed to hide the fact that there is no standardized method for creating dynamic libraries ( shared libraries ) in UNIX.
Instead of standardizing this method for all the Nixes — for example, implementing it as a single flag for the ld (1) command — with the participation of Peter Principle, this resulted in a separate competency of libtool. Moreover, in this case, this principle is noticeably well attached - the source code for devel / libtool contains 414,740 lines. Half of this code is tests, which is generally commendable, but in reality these are simply the consequences of Peter's principle: tests skillfully explore the functionality of a complex system in order to identify problems that should not have arisen at all.
Even more annoying, 31,085 of those lines of code are an unreadable clumsy shell script called configure . The idea is that this configure script performs approximately 200 automated tests so that the user is not burdened with manual disassembly with libtool. This pipets stupid idea, which was fully criticized back in the 80s, when she first appeared. Stupid because it allows the source code to pretend to ostensibly portability, hiding behind the configure script as a facade, instead of actually ensuring high portability.
It’s absurd that the configure idea survived.
In 1980, we saw several UNIX implementations: Cray-1s with support for 24-bit pointers, Amdahl UTS (UNIX for IBM mainframe), a whole bunch of more or less competent implementations of SysV + BSD from manufacturers of minicomputers , under-UNIX crafts from companies like Data General , even a full-fledged UNIX clone - Coherent from Mark Williams .
Configure the scripts of that time were written by hand and performed tasks like finding out - this is BSD or SysV-like UNIX, in order to copy the appropriate Makefile and, maybe, a couple .h to the right place. Later, configure scripts became more ambitious , which can be recognized from afar as a precedent for Peter's Principle. But instead of standardizing UNIX and eliminating the need for these scripts, some wise guy wrote the autoconf program to automatically generate configure scripts.
Today's UNIX / Posix-like systems, even if we take into account the IBM z / OS mainframe version, are practically the same from the point of view of the observer from 1980. Until now, 31,085 lines of the configure script code for libtool check for <sys / stat.h> and <stdlib.h> despite the fact that even those nixes that lacked them did not have enough memory to run libtool ; or disk space to fit 16 MB of libtool source code.
Well, for reasons that never bothered anyone, the autoconf program was written in the antediluvian macro language m4 , which led to the appearance of the tests:
Needless to say, adequate programmers would never wish to voluntarily delve into it, even if they had the skill, so these scripts for autoconf are created by copy-paste, often being lost among the overly fatty standard macros covering the “standard tests” that I mentioned above.
Yes, the very same tests that check for compatibility problems that no one has come across for 20 years.
It also explains why libtool checks for at least 26 different Fortran compiler names, which I simply don’t have on my system, and then spends another 26 tests to find out which of these 26 nonexistent compilers support the -g flag.
This is the sad reality of the bazaar, praised by Raymond in his book. A bunch of old rotten through hacks, endlessly copied by a generation of ignorant a la "IT professionals." You can shout “IT architecture!” Into their ears, but they still won't hear.
Today it is hard to believe, but under this debris that impedes movement are the ruins of a beautiful UNIX cathedral, deservedly famous for its simplicity of design, adequacy of functions, elegance of performance ... Everything is ashen, in other words.
One of Brooks’s magnificent sayings is: “Quality comes only when someone is personally responsible.” I’m surprised that Brooks doesn’t use UNIX as an example to this statement, because we can determine with almost surgical precision the moment when UNIX began to fragment; in the early 90s, AT & T singled out UNIX and sold all rights to it to Novell, thereby taking system of its architects. (Denis Ritchie compared this deal with the sale of the soul for a loaf of bread - translator's note ).
Others have recently come to the same opinion as Brooks. Some tried to impose a kind of sanity or even lay down the law formally, in the form of technical standards, hoping to restore order and structure in the bazaar. All attempts have failed spectacularly, because the generation of the dotcom-geeks lost in the bazaar has never seen the cathedral and, therefore, cannot even imagine why and why we need these cathedrals.
This is a sad irony, of course. Those to whom the book of Design of Design is devoted to Brooks will find it completely incomprehensible. Well, for those comrades who at least once thought that using m4 macros to configure autoconf to generate a shell script to check for the presence of 26 Fortran compilers to build a web browser is somehow a bit through the ass, Brooks suggested a well-reasoned hope that we have a chance to fix it.
--- end of translation ---
UPD: I receive advice on all channels to improve the translation and correct errors in the text, I express my gratitude to all those who took the time to do this. Thank.
~ Xlab
- Frederick F. Brooks
Hi, Habr!
')
I bring to your attention a free translation of Paul A-Henning Kamp's e- essay " A Generation Lost in the Bazaar ", telling us about the sad fate of a generation of IT professionals who grew up during the dotcom boom , as well as fundamental UNIX issues that directly affect software quality and portability . Everything in order.
Cathedral and Bazaar
In my article, the author uses the cathedral and bazaar metaphor described in Erik Raymond’s essay “ Cathedral and Bazaar ”, I find it appropriate to provide an excerpt from the text of a recognized translation of the essay:
--- start of quotation ---
Linux is an amazing system. Who would have thought that several thousand developers scattered all over the planet and collaborating only via the Internet would be able to create a world-class operating system. I did not think so at all. By the time Linux came to my attention in early 1993, I had been involved in the development of UNIX and open source software for about ten years.
Linux turned my ideas about what I know. I thought that the main thing in developing small tools for UNIX was their quick design and evolving programming over many years. And at the same time, I believed that as the complexity of the development increases, a more centralized approach is needed. I believed that developing the most complex software (for example, operating systems or simply large tools such as Emacs) should be like building a cathedral. Such programs should be created by individualistic masters or small groups of wizards working in strict isolation, avoiding premature beta versions.
I was very surprised by the development style of Linus Torvalds - the frequent release of releases, the availability of all source texts and the tolerance for heterogeneous programs. This is quite unlike the measured construction of the cathedral, the Linux community is more like a noisy bazaar, with many different approaches and directions. The fact that an agreed stable operating system is born on this market seems like a miracle of wonders.
I was shocked that this market style works and works well. I not only participated in the development of individual projects, but also tried to understand why in the Linux world not only confusion does not arise, but on the contrary, it moves forward at a speed that the cathedral builders can only envy.
By the middle of 1996, it seemed to me that I had begun to understand. Fate gave me a great chance to test my theory. It was an open source software development project that I deliberately spent in the marketplace style. The success of this project exceeded all expectations.
--- end of quotation ---
--- Start of translation ---
Introduction
Eric Raymond's book, The Cathedral and the Bazaar , published thirteen years ago, has expanded our vocabulary with new terminology. In addition, she predicted the end of the cascading software development model and the decline of the era of large software companies, thanks to the Free Software movement that had emerged among the people.
After reading the book I had something to think about, but she did not convince me. On the other hand, I have a direct relationship to the world of open source software, so even though I cannot help, it would be great if the author were right.
Another book that turned out to be in my house by the sea this summer also brings thoughts. She is clearly deeper than Raymond’s book (by the way, positively refers to him), this is Frederick P. Brooks's - “The Design of Design” . And how many times I nodded in agreement, enjoying Brooks’s writing style and his ability to submit material, the same time I was depressed by what I read.
Dotcom era
Thirteen years ago, the dot-com era of the 90s was reached, when every schoolchild was a web programmer or everyone who had been expelled from college already owned a web start-up. I got real pleasure when I tried to teach these salads with fundamental professional skills - creating automatic deployment scripts, version control systems, and so on. Nostalgia, of course - in fact, it was not fun.
We cannot deny that the whole dot-com era was a real disaster for the IT / CS industry in general and for UNIX in particular. The quality of software products was hit.
I do not have exact information on how much the IT industry has increased in volume during the period of the dot-com. My personal assessment is, if we count the number of professions that have surfaced, that our niche has grown by two orders of magnitude, that is, by more than 10,000% .
It's easy to fall in love with computers - almost anyone can make a work program, just like almost anyone can nail one board to another, in just a few attempts. The problem is that the share of such unprofessionally nailed boards is extremely small, if we consider the market of home furniture. And to move from two boards to a decent-looking set of chairs or a built-in wardrobe requires talent, practice and education. Those same (10 000% - 100%) = 9 900% additional percent had neither experience nor education when they entered the IT industry. And before they had a chance to get it, the party was over and most were left without work at all.
I will show mercy and note that those who clung to the position were undoubtedly the most talented and most skillful of all, but even in this case, it cannot be denied that they did not take place as IT professionals because of the lack of life experience.
The market style that E. Raymond so advocated, “ Just started drinking ”, opposed to well-designed cathedrals from the pre-Dotcom period, did not disappear after the dot-com crash, unfortunately, even today UNIX is sinking under its own weight.
UNIX
I updated my laptop. For 18 years now, I have a dev-version of FreeBSD on it and it takes all day to build a minimalist work environment from source, because it tries to build logic and architecture from anarchic garbage, called E. Raymond as a bazaar.
From afar, it seems that the FreeBSD ports collection is something like a map for a bazaar, so that FreeBSD users find it easier to find something. In practice, this “map” consists, at the moment, of 22,198 files with a brief description of each tent at the bazaar — a few lines about what is inside and where to read more about it. There are also 23 214 make-files telling us what to do with each software from those that we find in tents. These makefiles try to inform us about possible scenarios, various custom options and their default values. The card comes with 24,400 patch files to smooth out the curvature of the proposed handicrafts. Although, in fact, most of these patches are used to fix software portability issues.
Finally, this map helpfully tells us that if we want to install www / firefox , then first we need to deflate devel / nspr , security / nss , databases / sqlite3, and so on. As soon as we find them using the map, and then look for their dependencies, and then recursively get a list of dependencies of their dependencies, then we will have a list of purchases for 122 packages that we definitely need before we can get www / firefox .
We all love the component-oriented approach to software development. But even in the most trivial case, however, the code reuse methodology is completely alien in the market: the programs in the FreeBSD ports collection contain at least 1,342 copy-pastes of cryptographic algorithms.
If such a disregard for the reuse methodology would have been embodied as a mechanism of self-contained and independent software packages, then there would be a trade-off between code duplication and the ease of package management. But this is clearly not our case - the packets form an intricate web of unsystematic dependencies, which leads to even more duplication of code and a waste of resources.
Here's an ironic example: Sam Liffera's graphics / libtiff library is on its way to www / firefox , being one of those 122 packages in a common dependency chain. While the target program — the Firefox web browser — cannot display TIFF pictures.
. I did not try to figure out the reasons why 10 packages out of 122 require Perl and 7 require Python ; And one of them, devel / glib20 , requires both languages, I can not even imagine why.Next on the list ... here is Peter's Principle , which sounds like "In a hierarchical system, any worker rises to the level of his incompetence," and globally it can be formulated as: "Any well-working thing or idea will be used in increasingly difficult conditions until will cause a catastrophe. " In software engineering, a special case is used - “A dying project, which is gradually becoming too complicated for it to be understood by its own developers”. You see, we need 3 different versions of make, a macro processor (m4), an assembler, and a bunch of other interesting packages. At the end of the food chain, so to speak, is the libtool tool, which is designed to hide the fact that there is no standardized method for creating dynamic libraries ( shared libraries ) in UNIX.
Instead of standardizing this method for all the Nixes — for example, implementing it as a single flag for the ld (1) command — with the participation of Peter Principle, this resulted in a separate competency of libtool. Moreover, in this case, this principle is noticeably well attached - the source code for devel / libtool contains 414,740 lines. Half of this code is tests, which is generally commendable, but in reality these are simply the consequences of Peter's principle: tests skillfully explore the functionality of a complex system in order to identify problems that should not have arisen at all.
Even more annoying, 31,085 of those lines of code are an unreadable clumsy shell script called configure . The idea is that this configure script performs approximately 200 automated tests so that the user is not burdened with manual disassembly with libtool. This pipets stupid idea, which was fully criticized back in the 80s, when she first appeared. Stupid because it allows the source code to pretend to ostensibly portability, hiding behind the configure script as a facade, instead of actually ensuring high portability.
It’s absurd that the configure idea survived.
In 1980, we saw several UNIX implementations: Cray-1s with support for 24-bit pointers, Amdahl UTS (UNIX for IBM mainframe), a whole bunch of more or less competent implementations of SysV + BSD from manufacturers of minicomputers , under-UNIX crafts from companies like Data General , even a full-fledged UNIX clone - Coherent from Mark Williams .
Configure the scripts of that time were written by hand and performed tasks like finding out - this is BSD or SysV-like UNIX, in order to copy the appropriate Makefile and, maybe, a couple .h to the right place. Later, configure scripts became more ambitious , which can be recognized from afar as a precedent for Peter's Principle. But instead of standardizing UNIX and eliminating the need for these scripts, some wise guy wrote the autoconf program to automatically generate configure scripts.
Today's UNIX / Posix-like systems, even if we take into account the IBM z / OS mainframe version, are practically the same from the point of view of the observer from 1980. Until now, 31,085 lines of the configure script code for libtool check for <sys / stat.h> and <stdlib.h> despite the fact that even those nixes that lacked them did not have enough memory to run libtool ; or disk space to fit 16 MB of libtool source code.
How did this happen?
Well, for reasons that never bothered anyone, the autoconf program was written in the antediluvian macro language m4 , which led to the appearance of the tests:
Needless to say, adequate programmers would never wish to voluntarily delve into it, even if they had the skill, so these scripts for autoconf are created by copy-paste, often being lost among the overly fatty standard macros covering the “standard tests” that I mentioned above.
Yes, the very same tests that check for compatibility problems that no one has come across for 20 years.
It also explains why libtool checks for at least 26 different Fortran compiler names, which I simply don’t have on my system, and then spends another 26 tests to find out which of these 26 nonexistent compilers support the -g flag.
This is the sad reality of the bazaar, praised by Raymond in his book. A bunch of old rotten through hacks, endlessly copied by a generation of ignorant a la "IT professionals." You can shout “IT architecture!” Into their ears, but they still won't hear.
Today it is hard to believe, but under this debris that impedes movement are the ruins of a beautiful UNIX cathedral, deservedly famous for its simplicity of design, adequacy of functions, elegance of performance ... Everything is ashen, in other words.
One of Brooks’s magnificent sayings is: “Quality comes only when someone is personally responsible.” I’m surprised that Brooks doesn’t use UNIX as an example to this statement, because we can determine with almost surgical precision the moment when UNIX began to fragment; in the early 90s, AT & T singled out UNIX and sold all rights to it to Novell, thereby taking system of its architects. (Denis Ritchie compared this deal with the sale of the soul for a loaf of bread - translator's note ).
Others have recently come to the same opinion as Brooks. Some tried to impose a kind of sanity or even lay down the law formally, in the form of technical standards, hoping to restore order and structure in the bazaar. All attempts have failed spectacularly, because the generation of the dotcom-geeks lost in the bazaar has never seen the cathedral and, therefore, cannot even imagine why and why we need these cathedrals.
This is a sad irony, of course. Those to whom the book of Design of Design is devoted to Brooks will find it completely incomprehensible. Well, for those comrades who at least once thought that using m4 macros to configure autoconf to generate a shell script to check for the presence of 26 Fortran compilers to build a web browser is somehow a bit through the ass, Brooks suggested a well-reasoned hope that we have a chance to fix it.
--- end of translation ---
Chance to fix it
- "Ubuntu develops its own package format for installing third-party applications" ;
- The article "Qt Build System: a lifeline to build" - I recommend reading, there are many references to make criticism, good examples of working with QBS .
UPD: I receive advice on all channels to improve the translation and correct errors in the text, I express my gratitude to all those who took the time to do this. Thank.
~ Xlab
Source: https://habr.com/ru/post/183494/
All Articles