Python dependency management: it looks like you can already use it

In most popular programming languages ​​and ecosystems with dependencies everything is bad. As a rule, the creators of a new programming language do not pay much attention to this: simply because the new language does not yet have hundreds of thousands of libraries for different architectures and versions that depend on each other in a non-trivial way. And when these hundreds of thousands of libraries appear, it is already too late to change anything.

The only exception to my memory is node.js, the authors of which developed “from scratch” a surprisingly successful dependency management system. Well, how successful? There are also many problems there, starting with auto-executable scripts and ending with the transition from the tree structure to the flat structure in the 3rd version. But compared to what was at that time in other languages, the node is a breakthrough.

More recently, the ecosystem has been supplemented with a new rnpm utility that allows you to install React Native dependencies with one command. In which, for a moment, there may be a binary code for android and ios. For different architectures. And it all works out of the box. We at Voximplant are well acquainted with this thing: it puts our own React Native SDK .
')
We bring to your attention an interesting article published just two days ago, which tells in great detail about dependency management in Python. About the history of development. About the problems. And the most valuable thing is about how the community solves them. Under the cut adapted for Habr translation and the opportunity to discuss the sad topic of dependencies. And not just for Python.

Okay guys. It seems that those days are over when we could tell each other how bad things are with dependencies in Python. I'll tell you why.

Because we fixed the work with dependencies. If you are a developer and create or use Python libraries, now this is done easily and without pain. I say this because for a long time the Python tools for working with dependencies were ... with problems. This is no longer the case, but many people continue to read about "problems with dependencies in Python" and think that nothing has changed. It's time to tell everything.

A small historical excursion. With Python's dependencies, it was like this ...

Dawn

Python saw the light of the time when adding dependency meant a long journey to cyberspace. First you had to wait for the release of the telephone line. Then call the provider. Call on the modem, which is typical. After a battle with a SLIP or PPP client, you could ask a question in one of the groups, does anyone know a good gopher site where you can find a library to solve the problem you are facing. After that, it was possible to disconnect from the Internet and wait a few dozen hours: what if somebody answers. If you were lucky, you could put the download for the night.

There was no pip search then!

For that time, Python’s approach to dependency management was surprisingly shrewd. The 'import' directive and the extensible module loading system made it possible to connect dependencies from anywhere.

Along with version 2.0 of Python (October 2000), Distutils was released. With its help, developers were able to abstractly describe their collections of modules and create collections of modules and packages suitable for distribution. And again, it was a very, very shrewd decision. In those days, nothing of the kind was out of the question.

Rewinding time by 2004. Creating setuptools to solve common problems faced by open source developers when distributing their modules via the Internet. In 2005, the easy_install utility was added to automatically detect dependencies and download them to the necessary directories.

Dark Ages

Unfortunately, along with simple dependency description engines, setuptols dragged with them a monstrous amount of complexity. The author considered that the 'import' directive should work a little differently, and the setuptools installation changed its behavior. Compared to the usual 'import' directive, the setuptools version supported working with several versions of the same dependency within the same program. Then it seemed like a great idea. But time has shown that it is not . Moreover, the similar idea of ​​the possibility of installing several versions of a library on one computer (and not simultaneously for one program) is an important and necessary thing.

In addition to this ideological departure from the standard Python semantics, setuptools had problems with the maintainer. More precisely, with the lack of maintainers. It became a critical part of the Python ecosystem just at the moment when the author decided to take up other projects - completely unconnected with programming. For a long time no one could agree on who and how will support the project. Which, by that time, on the one hand, was already forknut , and on the other hand, in many distributions, the old and buggy version was “petrified”.

From 2008 to 2012, dependency management in Python was not the best show. It was at least painful to use. It was not clear which libraries and utilities to use, which ones it would be worth investing in and pay attention to. Manual dependency management was tedious, while automation required many poorly documented workarounds.

This is not to mention the numerous security holes in different parts of the tulchan. It was not possible to pack and distribute the Python library with native code so that the user would not need to have a fully configured compiler on his computer.

At the same time, new languages ​​and ecosystems began to gain more and more popularity, in which dependency management was implemented more or less correctly from the very beginning and which had better support for the distribution of binary files. These solutions learned their lesson from Python and Perl errors.

And finally, the Python Package Index. A site that stores and distributes all open source libraries created by the Python community. No more than a proof-of-concept that was launched too early, had no resources to develop its infrastructure and too often became offline .

This was not the best way for Python.

Interlude

Here we come to the point where they say that “dependency management in Python is # $ @ # @@!”. Most of these statements are outdated information describing this time period. A lot of blogposts with complaints, highly ranked by search engines for the query “problems”. Those who used Python at this time and switched to using other languages ​​often grumble about how awful it was to manage dependencies, how broken the ecosystem was and how often PyPI lay. Worst of all, the network has a lot of tips on workarounds to solve problems that have not been needed for a long time, but are still shown by search engines in the top of the list. Following to such councils breaks working environment to users and only aggravates a situation.

Risen from the ashes

And in the middle of this great break, a number of developers heroically, silently, slowly, all repaired, one by one biting into the bug reports. The 'pip' project was launched , whose numerous maintainers removed most of the complexity that 'easy_install' brought, and corrected many of its flaws. Donald Stufft immediately took over Pip and PyPI, fixing old vulnerabilities and making the system more resilient. Daniel Holth wrote a PEP for the 'wheel' format , which allows you to distribute binary code along with libraries. That is, in other words, allows users without a customized “C” compiler to install packages that would otherwise require this compiler to build.

In 2013, setuptools again merged with its distribute fork , which allowed operating system vendors to update their dependency systems and provide users with modern tools. The “ensurepip” module was included in the Python Core distribution for versions 2.7 and 3.3, which allowed users with a more or less modern version of Python to get a working dependency environment with one team.

New Renaissance

I will not talk about modern methods of working with dependencies. A separate website is dedicated to this. But I will show how simple it is now. For any modern version of Python, if you want to get a working environment without admin rights, all you need is:

$ python -m ensurepip --user $ python -m pip install --user --upgrade pip $ python -m pip install --user --upgrade virtualenv 

For each project, you can create your own sandbox virtualenv:

 $ python -m virtualenv lets-go $ . ./lets-go/bin/activate (lets-go) $ _ 

It's all. You can use pip to install any dependencies. To install most you don't even need a compiler! Moreover, these spells do not depend on the version of Python: they will work with Python 2, Python 3, PyPy and even Jython (translator's note: surprise, yes?).

In fact, often you don’t even need a step with 'ensurepip', since 'pip' will already be installed and configured! However, it is better to perform this simple spell, it will not harm.

More advanced operations on dependencies are also simplified. Need a "C" compiler ? The creators of the operating systems worked closely with the community to make the compiler setting as painless as possible:

 $ apt install build-essential python-dev # ubuntu $ xcode-select --install # macOS $ dnf install @development-tools python-devel # fedora C:\> REM windows C:\> start https://www.microsoft.com/en-us/download/details.aspx?id=44266 

Okay, okay, the last step is not so painless. But at least there is a compiler, it is free and is installed by simply downloading it from the site! Would you like to upload something to PyPI? For most projects, it will be that simple:

 $ pip install twine $ python setup.py sdist bdist_wheel $ twine upload dist/* 

Want to build binary wheels packages? And for this there is an application (docker container, of course).

What is important, you can confidently talk about high online PyPI. And they will have a new, completely reworked site from day to day ( translator's note: another surprise) .

I do a lot of Python development, and I can say that the most serious problem I encountered in the last year was the removal of cached files. I use modern toolchain every day, and it works!

The work that we have to

The current situation is not bad. But it can not be called "wonderful." What else would I like from our ecosystem:

• Tools for installation on end-user computers can be better.
  
  
• Pip does not damage the graphical user interface, so that it can be used without command line spells (translator's note: here Ostap suffered ...)
  
  
• You need tools to automatically create and update “setup.py”. Well, or something like “setup.python.json” or something like that, so that there is no need to write code just for a couple of lines of metadata.
  
  
• Error messages, when it is impossible to compile something with the compiler, should be more understandable and contain instructions for the user.
  
  
• PyPI should automatically build “wheels” for all platforms upon loading the sdists package. Yes, this is a big and ambitious task, but think how much it will simplify work with addictions!

I can go on for so long. There are many ways to improve dependency management in Python.

Afterword

What I wanted to say this longridom. The situation with dependencies in Python is far from ideal, but you can’t say that other programming languages ​​are “much better”. For Go, discussions continue on different options for managing dependencies and the situation with “CGo” is far from ideal . Node has its own hotly debated problems with dependency management culture and toolchain. Hackage is cool and all that, but the build time reaches truly astronomical values . As usual, with Rust and Cargo, everything is almost perfect - but after all, none of those who read these lines use Rust?

I do not want to say that working with dependencies in these languages ​​is somehow especially bad. In fact, they are very good, especially if we compare this with the average temperature in the hospital a couple of years ago; there is a steady progress and improvements visible to the end user.

In my opinion, any statement that in some language working with dependencies is much better than in Python is likely outdated. Now it is already possible to use dependencies in Python, and it doesn’t hurt .. Of course, it may be even better, but many programmers are working on improvement, and they have already successfully removed all the main barriers that prevented further development.

Use virtualenv! Hack yourself a little setup.py! If the last time you worked with dependencies in Python for a long time - try it now. I promise, everything has become much better.