Understand Open Source

  Open Source makes me feel like a masochist.
 I love him, but often working with him is a pain.
 (c) mine 

Once, maybe 10 years ago, it was fashionable to predict the end of the open source. The Internet was full of headlines like "Why Open Source Dies Soon" or "Why Open Source Will Never Win Paid Software." These are not exact headers. Rather, the impression that has been preserved since then.

Now we see that open source is blooming and smelling. Sometimes it smells good, sometimes it doesn't. But in general, is still gaining popularity and even somehow evolves. Actually, let's talk about this: about what Open Source is now, and where it will go next. I would like to see your comments on this in the comments.
')
Immediately I warn you that I am not an analytical agency. Therefore, I will not provide large-scale statistics with far-reaching conclusions based on it. Rather, I will confine myself to “far-reaching conclusions” based on my work with open source, on communicating with people who are also doing this, and on observing the activities of various companies in this field.

I have planned three posts that I will post if the topic seems interesting:

1. This post, where we talk about how to learn to understand Open Source
2. Models of using open source to achieve business goals
3. In the third post I will propose a non-strict classification of open source code by its suitability. By how convenient, efficient and useful it is to work with various side projects.

What is open source, I first learned, probably in the late nineties. The first time I assembled the Linux kernel from the source codes of the year in 2001. Until 2008, I was interested in the open source only as a “free” toolkit for home and work. I take the word “free” in quotation marks, because I haven’t considered the combination of “Open Source” and “paid software” to be opposite in meaning for a long time. But more on that later. In 2008, I first started reluctantly watching the trends in open source from inside the company I was working for. But really began to work with him less than two years ago. But he had already managed to swallow some dozens of different projects.

All this time, the myths associated with open sorts were debunked in my brain. And I even suspect that the real open source story has developed in such a way as to dispel the myths that were once true. That's just in reality, they collapsed much earlier than in my head.

By the way, a little questionnaire: how do you use open source? (See at the very bottom of the post. Habr does not know how to insert questionnaires in the text)

At first I looked at open source software very romantically. In fact, I believed in the one-time legends about “red-eyed enthusiasts,” who write code in the night with the sole purpose of making the world a better place. Of course, these people were. Even in my student dormitory there were guys who were picking up the Linux kernel and GNU applications. And some of them even poured their code into the appropriate repositories. I suspect that even now in the dormitories of technical universities, people at night are engaged not only in sex and gaming. It is interesting even what they are now picking. If 15 years ago the answer was obvious - Linux - now I’m almost sure that this is not Linux. Obviously, red-eyed enthusiasts still exist today. But their contribution to Open Source is negligible.

The first myth that I had to part with was the idea that the open source software does not make money. It was very easy to say goodbye to this idea, because there is no need to go far for examples. Take at least corporate versions of Linux distributions. For example, Red Hat Enterprise Linux (in common - RHEL). Commercial Linux distributions are an iron example of what you can earn on the open source. You can’t dismiss an argument like "If someone tries to earn money, it does not mean that the idea is popular and that you can earn a lot." Virtually every large western company pays for such distributions. And how much money can be earned on this is easy to understand from Red Hat financial statements or from a brief summary of its financial metrics .

The openness of the open source is the second myth that I said goodbye to. In an amicable way, this myth should be with some stretch more or less equivalent to the first. However, in the minds of people, these two delusions lived separately. My case was no exception. It's funny that in the modern world, the first myth, apparently, has already become obsolete. But the second is still flourishing. I think there are several reasons for this:

1. people think that you can only pay with money. Although if a person spends the night trying to make the “free” software work, he pays his own time. (Which, as you know, almost all the same money). The endless fussing with graphic libraries is the reason why I once stopped using Linux GUIs for home use (although now the situation with them probably has already improved);
2. overlooked by the real payer. This situation is often found on Habré. “I will never buy this expensive integrated boxed solution. I'd rather buy a cheap iron for a penny and put an open source on it. Everything will work in the same way and even better! ”The interesting point here is that it’s realistic that“ everything will work ”is not paid by the commentator himself, but by his employer. This is it - that is the employer paid the craftsman so that everything worked. And he could have dismissed the craftsman and with the freed money to approach the secretary: “Please order this black box. And when it comes, plug it in, please. Nothing more needs to be done. ” Of course, I'm exaggerating a bit here. And even more than that: in order for people to start costing cheaper solutions delivered with the service, a business needs to achieve certain dimensions. But here we come close to the following reason;
3. size matters. Everyday “everyday” experience deceptively hints that the open source is free. You wrote, for example, Hello World in C, compiled it with an open GCC compiler - and everything is fine with you. And what if you have several million lines of code? And if you want to enable non-trivial optimizations in the compiler? With a very high probability, you will run into trouble. By the way, you will have them not only with the “free” compiler. The problems will be regardless of which compiler you use - with open source or not. You will have to pay anyway. As soon as we start to do something big and - which is also important - non-trivial, we come to the fact that you have to pay for the toolkit. At my previous place of work, even a relatively small team of 20 developers encountered errors in the Linux kernel every couple of months. And each time new servers were bought, it was necessary to re-integrate all the same Linux with them. This is an illustration of the fact that if your business has a large scale, you need support. You can pay for it by purchasing a “solution” with support in the kit, or you can install a free distribution and buy support separately. However, the money you lay out.
   
   Here are a few more reasons why you have to pay for Open Source:
   
   1. bad documentation
   2. hardware or software platform incompatibility
   3. non-trivial user interface, to work with which you need to train staff
   
   Now the third myth. This time, it’s about how enthusiasts mostly work on open source. This misconception is associated with the first two and was chronologically dispelled simultaneously with them. If you carefully look at the popular Open Source projects, you can see that the most interesting and high-quality open source is made by large commercial companies. For example, Java is the brainchild of Sun Microsystems. The now popular LLVM compiler was conceived by Apple, and later Intel and Google joined its development. Irreplaceable in the modern world, Hadoop is almost entirely created by Yahoo! ( between 2006 and 2011, Yahoo! poured into Hadoop about 90 thousand lines of code, the closest "pursuer" - about 10 thousand ).
   
   
   
   
   
   By the way, about Yahoo! .. About 15 years ago almost every Internet user periodically used the search engine of this company. See how many people know Yahoo! now. At the end of the post on this topic there is a questionnaire.
   
   
   
   
   
   Above, I called only the main misconceptions that exist or existed regarding Open Source. There are a lot of smaller ones. But to discuss them is not so interesting. It is much more interesting to talk about how to understand the trends that govern the development of modern open source. If you learn to analyze the largest events in the world of open source software, then to understand the general picture, prejudices become unnecessary.
   
   Universal recipes here can not be. I can only share what I consider useful for myself. I will be glad to hear in the comments about what you rely on.
   
   So, how to learn to understand open source?
   
   Recipe 1. Open Source - it's almost always about money. If you see incomprehensible but powerful “movements” in the open source community, try to understand who is trying to make money from it. Let's look at an example.
   
   Less than a year ago, Microsoft announced the withdrawal of .NET Core in Open Source. Question: why?
   
   
   
   
   
   If you try to understand this with the help of a search engine, then you can link to this in the Google Top. From the article it follows that the goal is to bury Java. Another question pops up on Quora , where the most popular answer says the same thing.
   
   Well, let's say that the goal is really that. But how is Microsoft going to make money on it? We know that Microsoft is not a charity. While it remains unclear how the company wants to make money, we can assume that we have no answer.
   
   Looking further. We find the official answer of Microsoft . By the way, I managed to find it only thanks to Yandex, although I used the same queries as in Google.
   
   What does Microsoft itself tell us?
   
   
   
   
   
   Read the article. There are many correct and beautiful words about what the client will receive from the new initiative. But not a word about what the company wants to receive. Actually, there it could not be. Companies always focus on the customer. “All for the client, damn it!” And this is absolutely correct.
   
   I was not able to find a suitable explanation on the Internet (perhaps you were more fortunate in this matter). So we start to think independently.
   
   Microsoft itself tells us (see picture above) that .NET went to open source for the sake of cross-platform. Wonderful! Microsoft will not lie to us, just will not give the whole truth :). We look at other initiatives of the company in this area. We notice that Microsoft is a big apologist for HTML5. I will not give specific links here, because The company's activities in this area are multifaceted. Any search engine will easily show everything.
   
   
   
   
   
   We look further. We see that the company recently announced - through one of its daughters - the project JUniversal . This is a tool for translating Java code to the .NET platform.
   
   
   
   
   
   So, found that the company is strongly interested in cross-platform applications. And not only on .NET. But why didn't Microsoft rush to develop Java?
   
   Again, we recall that Microsoft is not for charity. Java is controlled by the wayward Oracle company. Making yourself dependent on the vagaries of Oracle is a big risk. Therefore, Microsoft focuses either on the tools that controls itself, or on those technologies that no one controls. Those. it seems like, the company really is not averse to eating a piece from Java. But this is not the goal itself. What then is the goal?
   
   The answer often flashes on the web pages that are devoted to cross-platform efforts of the office. If you repeat the search that I briefly described above, you will see that the goal is to make applications that developers write for Android and iOS also run on Windows / Windows Phone. Bingo! Add the last brick: Microsoft does not have a rich ecosystem around its mobile platform. But Google and Apple have such an ecosystem. So Microsoft wants to use a foreign ecosystem to lure users of two mobile giants. "Guys! All your favorite applications work on our system! Ida to us. Our platform is glamorous ”(for example).
   
   
   
   
   
   Did I make the right conclusion? I do not know. But at least, this is a hypothesis that can be tested by observing the company's further actions.
   
   Such logical construction in the analysis of trends in Open Source have to do often. Because never will soft-giants come and say openly: “You know, we're going to exploit someone else's ecosystem here.”
   
   Long get an example ... I hope that the good. We go further.
   
   Recipe 2. Thinking about where Open Source can go, look for analogies. In the world, truly new ideas are very rare. I sometimes find it helpful to think that open source is in a broad sense for many hundreds of years. What I mean?
   
   Legally, open source is a publication. About how thin. The code is also subject to copyright. The use of artistic or scientific publications can be licensed. The same can be done with the code. The technologies described in scientific papers can be patented. The algorithms implemented in the code are also (although not in all countries; and often under a completely different sauce).
   
   How can such analogies be used? As always, give an example.
   
   Let's take something not the most trivial. Let's talk about the role of the state.
   
   State profitable publication. Thanks to them, private ideas become public knowledge, which is a serious engine of the economy. Therefore, the state is trying in every way to "lure" these ideas out of the population. For example, one of the main crutches of modern capitalism was created - the patent system. Patents make the idea public, but at the same time protected from extraneous attacks. Its further use is regulated by licenses.
   
   So, in the world of publications, the state plays a huge role. And many publications that are interesting for the development of the economy are born in an academic environment that is supported by the same state. Which one could now put forward a hypothesis?
   
   That's right: the state plays a huge role in the development of Open Source. And the academic environment, too. This refers, first of all, to the American state. Russian for the development of Open Source has so far done little.
   
   Before testing the hypothesis, let's take a closer look at the situation. How are technical ideas born in an academic environment commercialized?
   
   So that scientists “do not pinch” their ideas, the state allows them to patent them and even gives them the priority right to commercialization (with some reservations that are not essential here).
   
   As a result of scientific work, the technology is born, protected by patents and covered with licenses. Then scientists - provided that the university allows it - set up a startup and begin to bring the technology to the product. If everything goes well, they make money from it.
   
   Here the important point is that scientists start a business, having in their hands the ready-made technology created by them and paid for by the state.
   
   Now what do we see regarding our hypothesis? And the fact that a huge amount of niche software with open source really was born in the walls of research organizations. An example from personal practice. This summer I attended the ISC'15 (International Supercomputing Conference) conference. Virtually all of the software presented at the conference is open and originates in academic circles. At first it was developed with state money, and then the developers, having left the laboratories with ready-made technologies, created commercial offices around them. This includes, for example, all the most popular MPI implementations and task schedulers .
   
   I am constantly confronted with open source, which came out of the academic environment. Often these are very serious projects. Take at least a tool for data visualization VisIt . 1.5 million lines of code. Developed by a very large American laboratory LLNL. It can be used both by itself and as a library for Python, C ++, Java. By the way, I recommend.
   
   
   
   
   
   
   
   
   
   Here is another example of the role of public money. "Shaggy" list of 2003 year. World famous Open Source projects sponsored by the state.
   
   So, drawing an analogy with other types of publications, it is easy to conclude that the state is a serious driver of open source.
   
   This post may be read by intellectual property experts. Maybe one of them knows where to get statistics on the amount of money that the US is investing in Open Source? And by the number of commercial companies that came out of the academic environment on the grounds of Open Source?
   
   However, there is one more recipe.
   
   Recipe 3. Periodically discard recipes. Or look for new ones. Or do not use at all. Open Source is rapidly evolving. What seemed fiction a couple of years ago, today is already a reality.
   
   This “recipe” cannot be illustrated by example because of the essence of the recipe. But I will say something anyway :)
   
   Today I can “google” or “naked” a Python library for almost any applied task. Those. in the world at some point a truly convenient tool appeared that gave an additional impetus to the development of the Open Source community. It has become very easy to develop elegant, small but effective tools. (Although limited in scope). Each of them separately is almost imperceptible and, most likely, will never enter any product. But all together they create the basis for a new type of social network, or something ... People post their snippets instead of photos of legs and food. A significant role in the development of this phenomenon belongs to GitHub.
   
   Convenient tools led to a small open source communism, or something ... People put their work on the network just because it would be nice if someone else uses them.
   
   That's all for now. In the next post we will talk about the purposes for which individuals and companies enter Open Source.
   
   Continued: "Understand Open Source: Usage Patterns"
   
   Two promised polls: