Python in a box - venv in python 3.3

Surely, most of those who develop or deploy Python applications use virtual environments. In particular through virtualenv , written by Ian Bicking.

The idea turned out to be so good and common that something similar is now present in Python 3.3 from the box as a module venv . It is almost the same as virtualenv, only slightly better .

How it works?

The main difference between venv is that it is built into the interpreter and can be processed even before loading the system modules. To do this, when defining the base directory with libraries, the following algorithm is used:

• in the directory with the interpreter or a higher level, a file with the name pyvenv.cfg ;
• if the file is found, the home key is searched for in it, the value of which will be the base directory;
• In the base directory, the system library is searched (by the special os.py marker);
• if something went wrong, everything is rolled back to the value that is hard-coded in the binary.

')
That's the whole essence of venv, everything else is already a wrapper over it.

How to create?

It's very simple, you need to call the venv module with the -m key, or use the pyvenv built-in script:

 pyvenv /path/to/new/venv 

The script will create the specified directory, along with all parent directories, if necessary, and build a virtual environment. This can be done in Windows, only the call will be a bit more verbose:

 c:\Python33\python -m venv /path/to/new/venv 

When creating, you can add various parameters, such as enabling system site-packages or using symlink instead of copying the interpreter.

Unlike virtualenv, the new venv requires that the directory being created does not exist, or is empty. This is probably done to prevent conflicts with existing files. This bug in python 3.3, in 3.4 already fixed. (Thank you, svetlov ).

How to use?

You can use the good old activation method through bin / activate (Scripts / activate in windows):

 cd /path/to/new/venv . bin/activate python3 some_script.py 

Or you may not use it ; you just need to call the interpreter from the environment and everything will work automatically:

 /path/to/new/venv/bin/python3 some_script.py 

This of course does not work for scripts that are run directly through #!/usr/bin/env python3 , for them you still need to, as before, do the activation. The solution is - about it just below.

Update

If your system has updated version of python, the virtual environment sometimes also needs to be updated.
Everything is simple - we call venv in the same way as creating an environment by adding the --upgrade key:

 pyvenv --upgrade /path/to/new/venv 

This will happen automatically if you use symlink, but if you want to make a fixation of the python version and libraries apart from isolation, I would recommend doing the update manually.

EnvBuilder extension

All work on creating the environment falls on the class venv.EnvBuilder , this class is written so that it can be expanded.

For example, you can distribute , pip and the necessary initial dependencies from requirements.txt when initializing the environment. It is better to leave more complicated logic on the conscience of more tools designed for this, such as buildout or make, but the initial setup can be done at the EnvBuilder level.

When creating the environment, the create(self, env_dir) method is used, in the source class it looks like this:

  def create(self, env_dir): env_dir = os.path.abspath(env_dir) context = self.ensure_directories(env_dir) self.create_configuration(context) self.setup_python(context) if not self.upgrade: self.setup_scripts(context) self.post_setup(context) 

The method describes the essence of the whole process: creating a directory ( ensure_directories ), configuration ( create_configuration ), adding python binaries ( setup_python ) and adding activation scripts ( setup_scripts ).

At the end, the post_setup hook is post_setup , into which you can add your actions. It can be seen that post_setup is executed only when creating the environment, and during - upgrade it will not be executed. This is easy to fix by adding another hook:

 class ImprovedEnvBuilder(venv.EnvBuilder): def create(self, env_dir): """Overwrite create method (add more hooks)""" env_dir = path.abspath(env_dir) context = self.ensure_directories(env_dir) self.create_configuration(context) self.setup_python(context) if not self.upgrade: self.setup_scripts(context) self.post_setup(context) else: self.post_upgrade(context) def post_upgrade(self, context): pass 

As parameters, when calling methods after ensure_directories , the context will be passed to an object containing all necessary information about the created environment in the form of attributes. For some reason, the documentation does not yet describe these keys, but you can easily understand everything yourself by looking at the code of the ensure_directories method in the base class. I will cite the most useful attributes:

• context.bin_path - path to the directory with binaries and executable scripts
• context.env_dir - path to the directory with the created environment,
• context.env_exe - the path to the binary inside the environment.

Accordingly, to run the python script inside the environment, you can do:

 import subprocess import venv class MyEnvBuilder(venv.EnvBuilder): def post_setup(self, context): script = '/path/to/some_script.py' subprocess.call([context.env_exe, script]) 

Executable scripts inside the venv

Let's return to the problem with executable scripts inside the virtual environment.

In virtualenv, it was enough for them to specify the interpreter via #/usr/bin/env python3 and use, remembering to do . bin/activate . bin/activate . If you are satisfied with this approach, then you can continue to use it in venv .

There is a new path. Inside EnvBuilder, the install_scripts(self, context, path) method is implemented, which automates the copying of scripts and binaries to the created environment. In path you must pass the path to a directory with nested subdirectories “common”, “nt”, “posix”, etc. In the subdirectory, in turn, put the necessary scripts or binaries. In “common” scripts for all platforms, in “nt” - for Windows, “posix” - for Linux, Mac OS X and other posix systems.

In addition, for text files, setting values ​​is performed. Out of the box are supported:

• __VENV_DIR__
• __VENV_NAME__
• __VENV_BIN_NAME__
• __VENV_PYTHON__

An example of a python script to run:

 #!__VENV_PYTHON__ import sys import my_module if __name__ == '__main__': sys.exit(my_module.run(sys.argv)) 

__VENV_PYTHON__ will be replaced with the full path to the python interpreter in the virtual environment.

After installing such a script via install_scripts , it can be launched without the need to activate the environment through bin / activate.

...

• Documentation for the venv module
• PEP-405 - Python Virtual Environments
• A small repository on github , in which I made an example of an EnvBuilder add-on auto-installing distribute, a pip, a dependency list from requirements.txt, and a set of executable scripts.