Mypy extension with plugins

Good afternoon friends. And we continue to increase the intensity of launching new courses and are already pleased to announce that at the end of April classes will start on the course “Web-developer in Python” . In this regard, we traditionally share the translation of useful material. Let's start.

It is known that Python is a language with dynamic typing. It's very easy to write DSL-like frameworks that are difficult to parse with static type checking tools. Despite this, with the latest functional mypy innovations, such as protocols and literal types , as well as basic metaclass support and descriptor support, we can get exact types more often, but it is still difficult to avoid false positives and other negative factors. To solve this problem and avoid the need to customize the type system for each framework, mypy supports a plugin system. Plugins are modules in Python that provide plugin hooks that mypy will call when checking the types of classes and functions that interact with the library or framework. Thus, it is possible to more precisely select the type of the returned function, which is otherwise extremely difficult to express, or to automatically generate some class methods to reflect the effects of the decorator. To learn more about the plug-in system architecture and see the full list of features, read the documentation .


')
Linked plugins for standard library

Mypy comes with default plugins for implementing basic functions and classes, as well as ctypes , contextlib and dataclasses . It also includes plugins for attrs (historically, this is the first third-party plugin written for mypy ). These plug-ins allow mypy to more accurately determine the types and correctly check the code for the type using these library functions. To show this with an example, take a look at the code snippet:

 from dataclasses import dataclass from typing import Generic, TypeVar @dataclass class TaggedVector(Generic[T]): data: List[T] tag: str position = TaggedVector([0, 0, 0], 'origin') 

Above, get_class_decorator_hook() is called when defining a class. This adds auto-create methods, including __init__() , to the function body. Mypy uses such a constructor to correctly calculate TaggedVector[int] as the type for position . As you can see from the example, plug-ins work even with generic classes.

Here is another code snippet:

 from contextlib import contextmanager @contextmanager def timer(title: str) -> Iterator[float]: ... with timer(9000) as tm: ... 

Here, get_function_hook() provides the exact return type for the decorator contextmanager , so that the calls to the decorated function can be checked against a specific type. Now mypy can recognize the error: the argument for timer() must be a string.

Combination of plug-ins and plugs

In addition to using dynamic Python functions, frameworks often face the problem of having large APIs. Mypy needs stub files for libraries to check the code that uses these libraries (only if the library does not contain inline annotations, which is less common). Spreading stubs for large frameworks using typeshed is not a common practice:

• Typeshed has a relatively slow release cycle (shipped with mypy ).
• Incomplete stubs can lead to false calls, which will be extremely difficult to avoid.
• Do not just combine plugs from different versions of typeshed .

Stub packages, introduced in PEP 561 , allow you to do the following:

• Developers can release stub packs as often as they like.
• Users who have not chosen to use the package will not see false positives.
• You can safely install custom versions of several different stub packages.

Moreover, pip allows you to combine different stubs for libraries and corresponding mypy plugins into one distribution. The plugs for the framework or the corresponding mypy plugin can be easily developed and put together in one distribution, which is extremely useful because the plugins fill in the missing or inaccurate definitions in the plugs.

The latest example of such a package is SQLAlchemy stubs and plugin , with the first public release of version 0.1, which was published some time ago on PyPI. Although this project is in the early Alpha version, we can safely use it in DropBox to improve type checking. The plugin understands the basic ORM declarations:

 from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import Column, Integer, String Base = declarative_base() class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) name = Column(String) 

In the code snippet above, the plugin uses get_dynamic_class_hook() to tell mypy that Base is a valid base class, even if it doesn't look like that. Then get_base_class_hook() is called to define the User, and adds several automatically generated attributes. Next we create an instance of the model:

user = User(id=42, name=42)

It is called get_function_hook() , so mypy may indicate an error: an integer value is received instead of a username.

Stubs define a Column as a generic descriptor, so that the model attributes get the correct types:

 id_col = User.id # Inferred type is "Column[int]" name = user.name # Inferred type is "Optional[str]" 

We welcome PRs that add more accurate types to stubs (progress for core modules is tracked here ).

Here are some of the pitfalls that we discovered while working on the plugs:

• Use __getattr__() to avoid false positives in the early stages when the stubs are not completed (this prevents mypy errors if there are no module attributes). You can also use this in __init__.py files if any sub- __init__.py are missing.
• Descriptors often help with a more precise type definition for custom access to attributes (as in the Column example, which we looked at above). Using descriptors is fine even if the actual implementation of the runtime uses a more complex mechanism, including the metaclass, for example.
• Do not hesitate to declare the framework classes as generic. Despite the fact that they are not such at runtime, this technique allows you to more accurately determine the type of some elements of the framework, whereas runtime errors can be easily circumvented . (We hope that frameworks will gradually add built-in support for generic types, obviously inheriting the corresponding classes from typing.Generic .)

Recently released mypy plugins

Already, there are several available plugins for popular Python frameworks. Apart from the above plugin for SQLAlchemy , other notable examples of packages with plugs and the mypy plug- in include plugs for Django and Zope interfaces. Now these projects are under active work.

Installing and connecting packages stubs and plug-ins

Use pip to install the plugin package for mypy and / or stubs into a virtual environment where mypy already stands:

  $ pip install sqlalchemy-stubs 

Mypy will automatically detect installed plugs. To connect installed plugins, include them directly in mypy.ini (or in the user configuration file):

 [mypy] plugins = sqlmypy, mypy_django_plugin.main 

Developing mypy plugins and writing stubs

If you want to develop a package of stubs and plugins for the framework you are using, you can use the sqlalchemy-stubs repository as a template. It includes the setup.py , infrastructure testing using data-driven tests, and an example of a plug-in class with a set of plug-in hooks. We recommend using stubgen to automatically generate stubs that come with mypy to start using them. Stubgen improved somewhat in mypy 0.670 .

Check out the documentation if you want to learn more about the mypy plugin system . You can also search the Internet for the source codes of the plug-ins that were mentioned in the article. If you have questions, you can ask them here .

On April 15, a free open webinar on the course will be held by Vladimir Filonov , one of the organizers of the Moscow Python community, sign up, it will be interesting. And now we are waiting for your comments on the translated material.