Longrid Warning: you have been warned, many letters.

The distribution of applications that were “free” from system-wide dependencies has been developing for a long time. Ubuntu is very, very actively promoting its snap, gnome is a flatpack. Both promise paradise and freedom from rpm / deb. Let's think about the problem they want to solve and the price they ask for solving this problem.

Libraries

No one in the modern world can write an application without using someone else's code. There are several reasons:

• Many libraries are so serious that writing their functionality from scratch is a daunting task. Examples - support for unicode, font rendering, math.
• Other libraries offer a rather modest set of functions, but written so well that writing at least as well is almost impossible. Standard programming language libraries, various implementations of libc, etc.
• The cost of work on interacting with someone else's code (which this section is devoted to) is often lower than the price of maintaining your code. The density of "bugs on lines of code" is likely to be comparable, and you should catch your "bugs" yourself. Other (popular) libraries are likely to be debugged and corrected by someone else’s hands.

The key is that even if the functionality of a single library can be written from principle, the total number of necessary functions (and dependencies) gives an almost exponential increase in the number of tasks that need to be solved by moving the time to start working on the code of the program itself. in the unattainable distance.

An example for understanding the scale of the drama: let's say your application accepts two lines of input as optional arguments and displays them together after normalization. If you are writing an industrial application (an application that looks like a "real"), then:

• You need a command line parser
• Which should accept unicode
• And, perhaps, give the user a hint that he was sealed in the name of the argument
• What requires phonetic comparison
• And maybe regular expressions
• In general, you will have to support not only Unicode, but also other locales, which requires a library to support locales and ALL of what people have come up with in the context of locales.
• Concatenation of strings with normalization is another application of a separate unicode library, you yourself do not implement this.
• Displaying (command line help, your result) is likely to require support for ncurses, a library that supports different terminals (text mode can be bypassed, but applications often use color features).
• Tests involve the use of a test framework, possibly a library for mocks.

It is clear that such a complexity for the problem of “two lines” is a rough overengineering, but as soon as you start doing something more, the idea of ​​“everything yourself (a)” begins to go beyond the limits of the visible and realizable.

What do you think, how many libraries do you need to guarantee curl http (s): // ...? Lot. You will use one, but the dependencies of your dependencies are your dependencies.

Copypaste & vendoring VS dynamic linking

Given that the use of libraries is inevitable, the use itself may vary in implementation. Please note that we have two important words "use" and "realization of use". What does "use" mean? In the roughest form - the ability to call the library code when necessary. But the implementation of this:

• We can copy the code that does the operations we need. In the form of a piece of code (copy & paste), as a separate module in a programming language (an object file for compiled languages), or as a separate module (for interpreted languages). Somewhere immediately next to it goes and "copy the library source file to its directory with the application." What problems does it create? The main, main problem is that we lose (forever) the connection with the original. Even if the author of the original library corrects the error, we will not know about it. Moreover, if we just copied the code, then the next person working on the program will not even be able to find out that this code is “alien”. In fact, we cut off the path in the "write from scratch" question and took someone else's. However, we cut only a piece, because if there are errors in this code (and they will not be there , they will be there ), then their correction will require the corrective to go and understand the essence of the problem to the very bottom. Even if the trial requires reading several hundred thousand lines of source code and hundreds of RFCs (as well as comments that implementations differ from RFCs), we have no other way. The key error in this place is that we have lost the information that this is someone else's code. The presence of comments in the file can help, but will require active and deep human participation, because if we write in the comment "taken from libfoobar, src / lib / foo.c version 364a51577f3782dbf8e5d2482ab941980357c492", then someone will have to see where libfoobar is located what is the version and what has changed from the previous version ". To simplify this process, we need machine-readable meta-information.
• If we accompany the “alien code” with meta-information and use programs to manage this code (instead of copy-paste), then this is called vending , that is, controlled inclusion of someone else's code in your code. Technically, vendoring can occur at the stage of source text, linking objects into an executable file, importing modules (in interpreters) from the application, or even dynamic linking with "its" version of the library (more on that later).
• Finally, we can perform dynamic linking during the application launch phase. For compiled languages, these are ordinary so's, for interpreted ones - the presence of a module in system-wide import. If it can import multiple applications, this is a shared library. If the application "brought its own module", then the library is "own", even if its interface implies a "shared library". For example, if an application uses "its" version of so's, regardless of whether it is different from the general one or not, then this is vendoring. And if the system is imported, then it is a shared library (shared library).

What is the difference between these methods? I will briefly present the arguments, they were briefly discussed in many articles. Each of these arguments remains valid despite the presence of neighboring counterarguments:

• Saving memory (online and on disk) for so-cons, reducing the size of the installed system. The more applications using the same so's, the greater the memory savings. Accordingly, back, the more "their" libraries brings the application, so it is "fatter."
• The dispute over who is monitoring the vulnerabilities is the system (providing library updates) or the author of the application (updating it on time).
• Resolving dependency conflicts (vendoring solves this problem as general libraries require attention and accuracy from all participants in the process, sometimes creating insurmountable difficulties), that legendary dll hell.
• New versions of libraries - either they appear at the request of the authors of the application, or by the decision of the authors of the distribution. In one case, the author can bring the new feature he needs, in the other case, the distribution can bring an improvement to an existing application by supporting something new in the library (for example, hidpi screens started to work correctly in all applications dynamically linked with qt / gtk libraries) .

All these questions were dealt with many times earlier. Instead, I want to focus on the social aspects of the “all my own” and “all common” watershed.

Social contract and power maintainers

Shared libraries are cooperation, power, and responsibility. People determining which shared libraries are available in the operating system dictate to software vendors which shared libraries they can use. Much software can use different libraries, and an indication of which version to use is left to the discretion of the linker (for compiled languages) or the dependency file handler (pip, bundler, etc). If all applications in the distribution kit are built with the same requirements, then grace comes: if there is an error in some library, the maintainer of this library updates the version, and the correction is automatically applied to all applications. Even if the application is released every two years, the fix in the conditional openssl will be applied within a week. If a particular OS made a decision to abandon the old protocol, some modifications (for example, the user interface), then these changes will also apply to everyone. Look & feel in the general style, which (perhaps) can be changed by the user once and for all. Is this not grace?

The power and the struggle for it

... This grace requires that all applications can work with the selected version of the library. And what if some application wants a very, very new function from the library, and all other applications do not want to use it, because, for example, this is not the LTS release of the library, i.e. is it not stable enough? But the distribution kit can refuse to switch to new versions "from the principle", because we promised users only bug fixes, and new versions only in the next OS version, which (like) will be released in half a year. And this causes resistance from the authors of the application. Who are you to tell me which versions to work with? I am the author, I see it. I need libfoobar 3.14-pre2 or older, not your ancient dull libfoobar 3.10.

... At this point, the author simply writes in the application requirements libfoobar>=3.14-pre2 . Maintainer takes and patches the requirement, plus removes the code that depended on this library. May be. Or simply refuses to accept the new version with this dependency, until this dependency (libfoobar 3.16) is in the new version of the distribution.

If the author really needs to use the new version of users (for example, because the author does not want to support the old version), then he is looking for workarounds to send the application to the user.

The same thing happens when there are several distributions, some newer, some older. Maintaining older distributions, testing with different libraries is difficult. So the option "ship with your libraries" appears almost immediately.

Community tragedy

This creates the prerequisites for the emergence of the tragedy of the communities:

• Each manufacturer (the author of the software) wants to ship as he needs. Adapting to someone else’s rules (versions) is a waste of effort and time, much less that there are many different distributions in the world.
• Users want newer versions.

In this case, the more applications come with their libraries, the less benefit from the system libraries. Remember the "grace"? The less she is “universal,” the less she is grace. If the shared library is used by 5 different applications out of 995 others, then the benefit of this library is 0.5%. It's a shame, yes. Moreover, it hurts all users, even those who, in principle, do not have an urgent need for a new feature - but if the application is available only in a vendor-like form, then the user does not have options.

It turns out that we have a global extremum: all applications use only shared libraries (maximum common grace, inconvenience from the authors of individual applications) or “every by itself” (a thick distribution, with a bunch of applications that may have unnoticed, but widely used vulnerabilities, eating a lot of memory, but the author of each application is convenient).

It is here that we come to the dispute rpm / deb VS snap / flatpack

Freedom or slavery?

Ubuntu is very, very much fighting for snap'y. GNOME is sure that the future belongs to flatpacks. Each of them is a framework for deeply individualistic applications. Any electrones that have with them not only the under-bonnet browser, but also the under-hood operating system. Your libc, your libssl, your regexps, your ncurses, etc. Only the core is common, i.e. in essence, this is the same containerized application, but for the desktop. Give each your own core, and you get an appliance in the form of a virtual. Add metadata - and you’ll get Docker's container.

Individualism of applications (authors of applications) is understandable, but who then stands for the common good? Local major improvement is compensated by a slight overall deterioration of the distribution multiplied by purely applications. If everyone makes their own local improvements, then the amount of impairment becomes greater benefit from the amount of improvements.

It would seem that in this place the creators of distributions should act as custodians of general interest. But...

Politics

Ubuntu is more dependent on Debian than Canonical would like (a company that makes Ubuntu). The value of Ubuntu is not in the efforts of the Ubuntu mainteners, but in the huge software repository coming from Debian in a form where all applications work well with the efforts of thousands of individual maintainers of the Debian distribution packages. Canonical adds on top of this its effort to polish the result - and for that is loved by some. Add a little bit of marketing and a fixed lifecycle that the Enterprise likes, and it’s a great product.

... which depends on the will of thousands of volunteers somewhere there.

That does not suit almost any commercial company. How to break this addiction? That's right by making your application bundle. The more of their applications, the less upstreams "upstreams" will offend the company. It is enough to recall the story when a vote in Debian about systemd buried upstart, developed by Canonical.

But to mainten a few tens of thousands of applications, some of which are your own space (erlang, go, perl, python, R, julia, etc), and some are monsters in the corresponding domain (browsers, emacs, tex, pacemaker, etc) heavy work. No wonder these are thousands of maintainers.

... And there is an idea. And, let, the authors of the applications are the main applications themselves. We'll give everyone in the sandbox, let them dig. Authors get freedom, Canonical - applications that are not dependent on Debian and which at least someone maynite free. Users get ...

... applications that are fat, heavy, which have irregular updates and that can easily keep vulnerabilities unpatched for years ... But some of them are shiny new.

So, what is next?

Imagine a world in which everyone takes everything with them ... Do you know what it looks like? Look at the chefsdk. He ships his postgresql (with his dependencies) with him, his rabbitmq (which depends on his erlang), plus the chef-server also on erlang, so he also has his erlang. Suddenly, we have two erlang and several dozen copies of the same libraries, slightly different in version, within one application. This is not the final version, because inside there are shared libraries between components. If we cut them further, we will get several dozen copies of openssl and libc into one application. Even not in the final form, it looks like 600MB per application.

... Which, of course, is multiple times larger than the average application for an electron. ... And 12 times more than the whole mariadb server (the whole DBMS!) Or krita or gimp (huge graphics applications).

And if everyone will be like this? I have 2000 packages installed on my computer (not counting -dev and lib) ... 2000 * 300 = 600GB (For the average size of the resulting, I took half from chefsdk, because not everyone is so terrible about dependencies). Now they occupy about 7 GB (including resources, such as documentation, editors textures, CAD program templates, etc.).

If this turns into 600GB, then is this a tragedy of the communities in its purest form? In each moment taken, we observe local optimization (and solving someone’s inconvenience), but together, the sum of these local optimizations reduces the overall optimality of the system. In my opinion, more than the local gain of each of the participants.

I understand why Canonical is pushing a snap. I understand this and do not approve.