How the virtual environment libraries work

Have you ever wondered how the virtual environment libraries work in Python? In this article, I propose to get acquainted with the main concept that all libraries use for environments, such as virtualenv, virtualenvwrapper, conda, pipenv.

Initially, in Python there was no built-in ability to create environments, and this feature was implemented as a hack. As it turned out, all libraries are based on a very simple feature of the python interpreter.

When Python starts the interpreter, it starts looking for a site-packages directory. The search starts from the parent directory regarding the physical location of the interpreter executable (python.exe). If the module folder is not found, then Python goes to the next level, and does so until the root directory is reached. In order to understand that this is a directory with modules, Python searches for the os module, which should be in the os.py file and is required for python to work.

Let's imagine that our interpreter is located at /usr/dev/lang/bin/python . Then the search paths will look like this:
')

 /usr/dev/lang/lib/python3.7/os.py /usr/dev/lib/python3.7/os.py /usr/lib/python3.7/os.py /lib/python3.7/os.py 

As you can see, Python adds a special prefix ( lib/python$VERSION/os.py ) to our path. As soon as the interpreter finds the first match (presence of the os.py file), it changes sys.prefix and sys.exec_prefix to this path (with the prefix removed). If for some reason no match is found, then the standard path is used which is compiled into the interpreter.

Now let's see how this is done by one of the oldest and most famous libraries - virtualenv.

 user@arb:/usr/home/test# virtualenv ENV Running virtualenv with interpreter /usr/bin/python3 New python executable in /usr/home/test/ENV/bin/python3 Also creating executable in /usr/home/test/ENV/bin/python Installing setuptools, pkg_resources, pip, wheel...done. 

After execution, it creates additional directories:

 user@arb:/usr/home/test/ENV# tree -L 3 . ├── bin │ ├── activate │ ├── activate.csh │ ├── activate.fish │ ├── activate_this.py │ ├── easy_install │ ├── easy_install-3.7 │ ├── pip │ ├── pip3 │ ├── pip3.7 │ ├── python │ ├── python-config │ ├── python3 -> python │ ├── python3.7 -> python │ └── wheel ├── include │ └── python3.7m -> /usr/include/python3.7m ├── lib │ └── python3.7 │ ├── __future__.py -> /usr/lib/python3.7/__future__.py │ ├── __pycache__ │ ├── _bootlocale.py -> /usr/lib/python3.7/_bootlocale.py │ ├── _collections_abc.py -> /usr/lib/python3.7/_collections_abc.py │ ├── _dummy_thread.py -> /usr/lib/python3.7/_dummy_thread.py │ ├── _weakrefset.py -> /usr/lib/python3.7/_weakrefset.py │ ├── abc.py -> /usr/lib/python3.7/abc.py │ ├── base64.py -> /usr/lib/python3.7/base64.py │ ├── bisect.py -> /usr/lib/python3.7/bisect.py │ ├── codecs.py -> /usr/lib/python3.7/codecs.py │ ├── collections -> /usr/lib/python3.7/collections │ ├── config-3.7m-darwin -> /usr/lib/python3.7/config-3.7m-darwin │ ├── copy.py -> /usr/lib/python3.7/copy.py │ ├── copyreg.py -> /usr/lib/python3.7/copyreg.py │ ├── distutils │ ├── encodings -> /usr/lib/python3.7/encodings │ ├── enum.py -> /usr/lib/python3.7/enum.py │ ├── fnmatch.py -> /usr/lib/python3.7/fnmatch.py │ ├── functools.py -> /usr/lib/python3.7/functools.py │ ├── genericpath.py -> /usr/lib/python3.7/genericpath.py │ ├── hashlib.py -> /usr/lib/python3.7/hashlib.py │ ├── heapq.py -> /usr/lib/python3.7/heapq.py │ ├── hmac.py -> /usr/lib/python3.7/hmac.py │ ├── imp.py -> /usr/lib/python3.7/imp.py │ ├── importlib -> /usr/lib/python3.7/importlib │ ├── io.py -> /usr/lib/python3.7/io.py │ ├── keyword.py -> /usr/lib/python3.7/keyword.py │ ├── lib-dynload -> /usr/lib/python3.7/lib-dynload │ ├── linecache.py -> /usr/lib/python3.7/linecache.py │ ├── locale.py -> /usr/lib/python3.7/locale.py │ ├── no-global-site-packages.txt │ ├── ntpath.py -> /usr/lib/python3.7/ntpath.py │ ├── operator.py -> /usr/lib/python3.7/operator.py │ ├── orig-prefix.txt │ ├── os.py -> /usr/lib/python3.7/os.py │ ├── posixpath.py -> /usr/lib/python3.7/posixpath.py │ ├── random.py -> /usr/lib/python3.7/random.py │ ├── re.py -> /usr/lib/python3.7/re.py │ ├── readline.so -> /usr/lib/python3.7/lib-dynload/readline.cpython-37m-darwin.so │ ├── reprlib.py -> /usr/lib/python3.7/reprlib.py │ ├── rlcompleter.py -> /usr/lib/python3.7/rlcompleter.py │ ├── shutil.py -> /usr/lib/python3.7/shutil.py │ ├── site-packages │ ├── site.py │ ├── sre_compile.py -> /usr/lib/python3.7/sre_compile.py │ ├── sre_constants.py -> /usr/lib/python3.7/sre_constants.py │ ├── sre_parse.py -> /usr/lib/python3.7/sre_parse.py │ ├── stat.py -> /usr/lib/python3.7/stat.py │ ├── struct.py -> /usr/lib/python3.7/struct.py │ ├── tarfile.py -> /usr/lib/python3.7/tarfile.py │ ├── tempfile.py -> /usr/lib/python3.7/tempfile.py │ ├── token.py -> /usr/lib/python3.7/token.py │ ├── tokenize.py -> /usr/lib/python3.7/tokenize.py │ ├── types.py -> /usr/lib/python3.7/types.py │ ├── warnings.py -> /usr/lib/python3.7/warnings.py │ └── weakref.py -> /usr/lib/python3.7/weakref.py └── pip-selfcheck.json 

As you can see, the virtual environment was created by copying the Python binary into a local folder (ENV / bin / python). We can also notice that the parent folder contains symbolic links to the standard python library files. We cannot create a symbolic link to the executable file, since the interpreter will still name it to the actual path.

Now let's activate our environment:

 user@arb:/usr/home/test# source ENV/bin/activate 

This command changes the $ PATH environment variable so that the python command points to our local version of python. This is achieved by substituting the local path of the bin folder to the beginning of the $ PATH line so that the local path takes precedence over all paths to the right.

 export "/usr/home/test/ENV/bin:$PATH" echo $PATH 

If you run the script from this environment, it will be executed using the binary at /usr/home/test/ENV/bin/python . The interpreter will use this path as a starting point for finding modules. In our case, the modules of the standard library will be found on the path /usr/home/test/ENV/lib/python3.7/ .

This is the main hack, thanks to which all libraries work with virtual environments.

Python 3 enhancements

Starting with Python 3.3, a new standard has emerged, referred to as PEP 405 , which introduces a new mechanism for lightweight environments.

This PEP adds an extra step to the search process. If you create the configuration file pyenv.cfg , then instead of copying the Python binary and all its modules, you can simply specify their location in this config.

This feature is actively used by the standard venv module, which appeared in Python 3.

 user@arb:/usr/home/test2# python3 -m venv ENV user@arb:/usr/home/test2# tree -L 3 . └── ENV ├── bin │ ├── activate │ ├── activate.csh │ ├── activate.fish │ ├── easy_install │ ├── easy_install-3.7 │ ├── pip │ ├── pip3 │ ├── pip3.5 │ ├── python -> python3 │ └── python3 -> /usr/bin/python3 ├── include ├── lib │ └── python3.7 ├── lib64 -> lib ├── pyvenv.cfg └── share └── python-wheels 

 user@arb:/usr/home/test2# cat ENV/pyvenv.cfg home = /usr/bin include-system-site-packages = false version = 3.7.0 user@arb:/usr/home/test2# readlink ENV/bin/python3 /usr/bin/python3 

Thanks to this configuration, instead of copying the binary, venv simply creates a link to it. If the include-system-site-packages parameter is changed to true , then all modules of the standard library will be automatically accessible from the virtual environment.

Despite these changes, most third-party libraries for working with virtual environments use the old approach.

PS: I am the author of this article, you can ask any questions.