Python on wheels

The package system infrastructure for Python has long been criticized from both developers and system administrators. For a long time, even the community itself could not come to an agreement about which tools to use in each particular case. Already there are distutils, setuptools, distribute, distutils2 as basic distribution mechanisms and virtualenv, buildout, easy_install and pip as high-level tools for managing all this mess.

Prior to setuptools, the primary distribution format was source files or some binary MSI distributions for Windows. Under Linux, there were initially broken bdist_dumb and bdist_rpm , which worked only on systems based on Red Hat. But even bdist_rpm did not work well enough for people to start using it.

A few years ago, PJE tried to fix this problem by providing a mix of setuptools and pkg_resources to improve distutils and add metadata to the Python packages. In addition to this, he wrote the easy_install utility to install them. Due to the lack of a distribution format supporting metadata, the format of 'eggs' [egg] was provided.
')
Python eggs - the usual zip-archives containing the python-package and the necessary metadata. Although many people probably never deliberately collected eggs, their metadata format is still alive and well. And everyone expands their projects using setuptools.

Unfortunately, some time later, the community divided, and part of it proclaimed the death of binary formats and 'eggs' in particular. After this pip, replacing easy_install, stopped accepting egg-format.

Then a little more time passed, and the rejection of binary packages began to cause inconvenience. People have become more and more deployed to cloud servers, and the need to recompile C-shnyh libraries on each machine is not too happy. Since the “eggs” at that time were obscure (I suppose), they were altered in the new PEPs, and were called ' wheels ' [wheels].

In the following, it is assumed that all actions take place in a virtualenv environment .

What kind of wheel?

Let's start with the simple. What are the 'wheels' and how do they differ from the 'eggs'? Both formats are zip files. The main difference is that the egg can be imported without unpacking, the wheel will have to be unpacked. Although there are no technical reasons that make the 'wheels' non-imported, support for their direct import has never even been planned.

Another difference is that the eggs contain compiled bytecode, and the “wheels” do not. The main advantage of this is that there is no need to create separate wheels for each version of Python until you have to distribute modules linked through libpython. Although in new versions of Python 3, using stable ABI, even this can already be done.

However, the wheel-format is also not without problems, some of which it inherits from the 'eggs'. For example, binary distributions for Linux are still unacceptable for most because of two drawbacks: Python itself compiles for Linux in different forms, and modules are linked with different system libraries. The first problem is caused by the coexistence of incompatible versions of Python 2: USC2 and USC4. Depending on the compilation mode, the ABI changes. Currently, wheel (as far as I can tell) does not contain information about which Unicode mode the library is associated with. A separate problem is that Linux distributions are less compatible with each other than we would like, and circumstances may be such that an assembly compiled for one distribution will not work for the others.

All this translates into the fact that, generally speaking, at the moment binary 'wheels' cannot be loaded onto PyPI as incompatible with different systems.

In addition to all this, wheel now knows only two extremes: binary packages and packages containing pure python code. Binary packages are specific to Python 2.x branches. This does not seem to be a big problem now, because the 2.x cycle is coming to an end, and the packets collected for only 2.7 will last long. But if suddenly it was about Python 2.8, it would be interesting to say that this package does not depend on the version of Python, but it contains binaries, so it can not depend on the architecture.

The only case that justifies the existence of such a package is when it contains distributed libraries loaded with ctypes from CFFI. Such libraries are not connected through libpython and are not dependent on the implementation of the language (they can even be used with pypy).

But there is a bright side: nothing prohibits the use of binary wheels in their own homogeneous infrastructures.

Wheel assembly

So now we know what a wheel is. How to make your own 'wheel'? Building from your own libraries is the simplest process. All you need is the latest version of setuptools and the wheel library. Once both are installed, the 'wheel' is assembled with the following command:

 $ python setup.py bdist_wheel 

Wheel will be created in the package directory. However, there is one thing to be aware of: the distribution of binaries. By default, the collected 'wheel' (provided that no binary steps are used in setup.py) consists of pure-python code. This means that even if you distribute .so , .dylib or .dll as part of your package, the resulting 'wheel' will look platform-independent.

The solution to this problem is to manually implement Distribution from setuptools, removing the purity flag to false :

 import os from setuptools import setup from setuptools.dist import Distribution class BinaryDistribution(Distribution): def is_pure(self): return False setup( ..., include_package_data=True, distclass=BinaryDistribution, ) 

Wheel installation

Using the latest version of pip, the 'wheel' is set as follows:

 $ pip install package-1.0-cp27-none-macosx_10_7_intel.whl 

But what about addictions? Here there are some difficulties. Usually one of the requirements for the package is the ability to install it even without an internet connection. Fortunately, pip allows you to disable booting from the index and set the directory containing everything needed for installation. If we have wheels for all dependencies of the required versions, you can do the following:

 $ pip install --no-index --find-links=path/to/wheels package==1.0 

Thus, version 1.0 of the package will be installed in our virtual environment.

Wheels for addictions

OK, but what if we don't have .whl for all our dependencies? Pip in theory solves this problem using the command wheel . This should work something like this:

 pip wheel --wheel-dir=path/to/wheels package==1.0 

This command will download all the packages that our package depends on to the specified folder. But there are a couple of problems.
The first is that the team currently has a bug that does not unload dependencies, which are already 'wheels'. So if the dependency is already available on PyPI in a wheel-format, it will not be loaded.

This is temporarily solved by the shell script, which manually moves the downloaded wheels from the cache.

 #!/bin/sh WHEEL_DIR=path/to/wheels DOWNLOAD_CACHE_DIR=path/to/cache rm -rf $DOWNLOAD_CACHE_DIR mkdir -p $DOWNLOAD_CACHE_DIR pip wheel --use-wheel -w "$WHEEL_DIR" -f "$WHEEL_DIR" \ --download-cache "$DOWNLOAD_CACHE_DIR" package==1.0 for x in "$DOWNLOAD_CACHE_DIR/"*.whl; do mv "$x" "$WHEEL_DIR/${x##*%2F}" done 

The second problem is a bit more serious: how will pip find our own package if it is not on PyPI? Right, no way. Documentation in this case recommends using not the pip wheel package , but the pip wheel -r requirements.txt , where the requirements.txt contains all the necessary dependencies.

Build packages using DevPI

Such a temporary solution to the dependency problem is quite applicable in simple situations, but what if there are many internal python packages that depend on each other? This design is falling apart quickly.

Fortunately, last year Holker Krekel created a solution to this trouble called DevPI , which is essentially a hack that emulates the work of pip with PyPI. Once installed on a computer, DevPI works as a transparent proxy in front of PyPI and allows pip to install packages from the local repository. In addition, all packages downloaded from PyPI are automatically cached, so even if you turn off the network, these packages will be available for installation. And in the end, it becomes possible to upload your own packages to the local server in order to refer to them in the same way as to those stored in the public index.

I recommend installing DevPI in a local virtualenv, then adding links to devpi-server and devpi in PATH .

 $ virtualenv devpi-venv $ devpi-venv/bin/pip install --upgrade pip wheel setuptools devpi $ ln -s `pwd`/devpi-venv/bin/devpi ~/.local/bin $ ln -s `pwd`/devpi-venv/bin/devpi-server ~/.local/bin 

After that, it remains just to start the devpi-server , and it will work until manual stop.

 $ devpi-server --start 

After launching it, it is necessary to initialize it once:

 $ devpi use http://localhost:3141 $ devpi user -c $USER password= $ devpi login $USER --password= $ devpi index -c yourproject 

Since I use DevPI 'for myself', the DevPI username and system username are the same. At the last step, an index is created by the name of the project (if necessary, you can create several).

To redirect pip to the local repository, you can export the environment variable:

 $ export PIP_INDEX_URL=http://localhost:3141/$USER/yourproject/+simple/ 

I will place this command in the postactivate script of my virtualenv to prevent accidental loading from the wrong index.

To place your own wheels in the local DevPI, the devpi utility is devpi :

 $ devpi use yourproject $ devpi upload --no-vcs --formats=bdist_wheel 

The --no-vcs disables magic, which attempts to detect a version control system and moves some files first. I don’t need this, because in my projects I distribute files that I don’t include in VCS (binaries, for example).

Finally, I highly recommend splitting the setup.py files in such a way that PyPI will reject them, and DevPI will accept them so that they do not accidentally release their code using setup.py resease . The easiest way to do this is to add the wrong PyPI classifier:

 setup( ... classifier=['Private :: Do Not Upload'], ) 

Wrapping up

Now everything is ready to start using internal dependencies and build your own 'wheels'. Once they appear, they can be archived, downloaded to another server and installed in a separate virtualenv.
The whole process will become a bit easier when the pip wheel stops ignoring the existing wheel packages. In the meantime, the above shell script is not the worst solution.

In comparison with the 'eggs'

Now wheel format is more attractive than egg. Its development is more active, PyPI began to add its support and, as utilities start working with it, it looks like a better solution. 'Eggs' is currently supported only by easy_install, although most have long since switched to pip.

I believe that the Zope community is still the largest one based on egg-format and buildout. And I believe that if a solution based on 'eggs' is applicable in your case, it is worth applying it. I know that many do not use eggs at all, preferring to create virtualenv-s, archive them and send them to different servers. Just for this deployment wheels - the best solution, since different servers may have different paths to libraries. There was a problem related to the fact that .pyc files were created on the build server for virtualenv, and these files contain specific paths to the files. Using wheel, .pyc created after installation in a virtual environment and will automatically have the correct path.

So now you have it. Python on wheels. And it even seems to work, and perhaps worth the time spent.