Merge multiple packages into one Python namespace
Sometimes it becomes necessary to separate several packages lying in the same namespace along different physical paths. For example, if you want to be able to transfer different layouts of plug-ins, having the ability to later add them without controlling their location, and, at the same time, access them through one namespace.
This cheat sheet, which is more suitable for beginners, is dedicated to Python namespaces.
Let's take a look at how this can be done in different versions of Python, since although Python2 is no longer supported, many of us are right between two fires now, and this is just one of the important nuances in the transition.
')
Consider this example:
We want to get the package structure:
The contents of the file module1
The contents of the file module2
In this case, the packages are distributed in the following folder structure:
Suppose that one way or another, path1 and path2 have already been added to sys.path. We need to access module1 and module2:
What happens in Python 3.7 when this code is executed? Everything works wonderfully:
With PEP-420 in Python 3.3, support for implicit namespaces has been introduced. In addition, when importing a package from the py33 version, it is not necessary to create __init__.py files. And when importing a namespace, it’s just _prohibited_. If the file __init__.py is present in one or both directories and the name namespace1, an error will occur on the import of the second package.
Thus, the presence of an inishnik explicitly defines a package, and packages cannot be combined, it is a single entity. If you start a new, independent of the old development project and packages will be installed using pip, then you need to follow this method. However, sometimes we inherit the old code, which also needs to be maintained, at least for a while, or transferred to the new version.
Let's go to Python 2.7 . With this version it is more interesting, you must first add __init__.py to each directory to create packages, otherwise the interpreter simply does not recognize the package in this set of files. And then register explicitly the namespace in the __init__ files related to namespace1, otherwise, only the first package will be imported.
What happens when this happens? When the interpreter reaches the first import, a sys.path package with that name is searched, it is in path1 / namespace1, and the interpreter executes path1 / namespace1 / __ init__.py. Further search is not conducted. However, the extend_path function itself searches already around the sys.path, finds all packages named namespace1 and the initial and adds them to the variable __path__ of the package namespace1, which is used to search for child packages in this namespace.
In the official guides, it is recommended that the initials be the same with each placement of namespace1. In fact, they can be empty everything except the first one, which is located in the search in sys.path, in which there must be a call to pkgutil.extend_path, because the others are not executed. But, of course, it is better that the actual initiator be truly challenged, so as not to tie up his logic “on occasion” and not guess which initiator was executed first, because the search order may change. For the same reason, you should not have any other logic __init__ files in the field of variables.
This will work in later versions and this code can be used to write compatible code , but you need to take into account that the chosen method must be followed in each distributed package. If on the 3rd version we put an initial in some packages by calling pkgutil.extend_path, and some are left without an initial, it will not work.
In addition, this option is suitable for the case when you plan to install using python setup.py install.
Another way, which is now considered somewhat outdated, but you can still find it a lot where:
The pkg_resources module comes with the setuptools package. Here the meaning is the same as in pkgutil - it is necessary that every __init__ file at each location namespace1 contains the same namespace declaration and there is no other code. At the same time in setup.py you need to register the namespace namespace_packages = ['namespace1']. A more detailed description of creating packages is beyond the scope of this article.
In addition, you can often find such a code
Here the logic is simple - if setuptools is not installed, then we use pkgutil, which is included in the standard library.
If you configure one of these methods namespace, then from one module you can call another. For example, change the namespace1 / package2 / module2
And then we'll see what happens if we mistakenly named a new package as well as an existing one and wrapped it with the same namespace. For example, there will be two packages in different places called package1.
In this case, only the first will be imported and there will be no access to module2. Packages cannot be combined.
Summary:
Sources:
Examples can be found here .
This cheat sheet, which is more suitable for beginners, is dedicated to Python namespaces.
Let's take a look at how this can be done in different versions of Python, since although Python2 is no longer supported, many of us are right between two fires now, and this is just one of the important nuances in the transition.
')
Consider this example:
We want to get the package structure:
namespace1 package1 module1 package2 module2 The contents of the file module1
print('package 1') var1 = 1 The contents of the file module2
print('package 2') var2 = 2 In this case, the packages are distributed in the following folder structure:
path1 namespace1 package1 module1 path2 namespace1 package2 module2 Suppose that one way or another, path1 and path2 have already been added to sys.path. We need to access module1 and module2:
from namespace1.package1 import module1 from namespace1.package2 import module2 What happens in Python 3.7 when this code is executed? Everything works wonderfully:
package 1 package 2 With PEP-420 in Python 3.3, support for implicit namespaces has been introduced. In addition, when importing a package from the py33 version, it is not necessary to create __init__.py files. And when importing a namespace, it’s just _prohibited_. If the file __init__.py is present in one or both directories and the name namespace1, an error will occur on the import of the second package.
ModuleNotFoundError: No module named 'namespace1.package2' Thus, the presence of an inishnik explicitly defines a package, and packages cannot be combined, it is a single entity. If you start a new, independent of the old development project and packages will be installed using pip, then you need to follow this method. However, sometimes we inherit the old code, which also needs to be maintained, at least for a while, or transferred to the new version.
Let's go to Python 2.7 . With this version it is more interesting, you must first add __init__.py to each directory to create packages, otherwise the interpreter simply does not recognize the package in this set of files. And then register explicitly the namespace in the __init__ files related to namespace1, otherwise, only the first package will be imported.
from pkgutil import extend_path __path__ = extend_path(__path__, __name__) What happens when this happens? When the interpreter reaches the first import, a sys.path package with that name is searched, it is in path1 / namespace1, and the interpreter executes path1 / namespace1 / __ init__.py. Further search is not conducted. However, the extend_path function itself searches already around the sys.path, finds all packages named namespace1 and the initial and adds them to the variable __path__ of the package namespace1, which is used to search for child packages in this namespace.
In the official guides, it is recommended that the initials be the same with each placement of namespace1. In fact, they can be empty everything except the first one, which is located in the search in sys.path, in which there must be a call to pkgutil.extend_path, because the others are not executed. But, of course, it is better that the actual initiator be truly challenged, so as not to tie up his logic “on occasion” and not guess which initiator was executed first, because the search order may change. For the same reason, you should not have any other logic __init__ files in the field of variables.
This will work in later versions and this code can be used to write compatible code , but you need to take into account that the chosen method must be followed in each distributed package. If on the 3rd version we put an initial in some packages by calling pkgutil.extend_path, and some are left without an initial, it will not work.
In addition, this option is suitable for the case when you plan to install using python setup.py install.
Another way, which is now considered somewhat outdated, but you can still find it a lot where:
#namespace1/__init__.py __import__('pkg_resources').declare_namespace(__name__) The pkg_resources module comes with the setuptools package. Here the meaning is the same as in pkgutil - it is necessary that every __init__ file at each location namespace1 contains the same namespace declaration and there is no other code. At the same time in setup.py you need to register the namespace namespace_packages = ['namespace1']. A more detailed description of creating packages is beyond the scope of this article.
In addition, you can often find such a code
try: __import__('pkg_resources').declare_namespace(__name__) except: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) Here the logic is simple - if setuptools is not installed, then we use pkgutil, which is included in the standard library.
If you configure one of these methods namespace, then from one module you can call another. For example, change the namespace1 / package2 / module2
import namespace1.package1.module1 print(var1) And then we'll see what happens if we mistakenly named a new package as well as an existing one and wrapped it with the same namespace. For example, there will be two packages in different places called package1.
namespace1 package1 module1 package1 module2 In this case, only the first will be imported and there will be no access to module2. Packages cannot be combined.
from namespace1.package1 import module1 from namespace1.package1 import module2 #>>ImportError: cannot import name module2 Summary:
- In the case of Python over 3.3 and installation with pip, it is recommended to use an implicit namespace declaration.
- In the case of support for versions 2 and 3, as well as installations with both pip and python setup.py install, the option with pkgutil is recommended.
- The pkg_resources option is recommended if you need to support old packages using this method, or you need the package to be zip-safe.
Sources:
- packaging.python.org/guides/packaging-namespace-packages
- www.python.org/dev/peps/pep-0420
- Learning Python, 5th Edition by Mark Lutz
Examples can be found here .
Source: https://habr.com/ru/post/458432/
All Articles