Python: metaprogramming in production. Part one

Many people think that metaprogramming in Python unnecessarily complicates the code, but if you use it correctly, you can quickly and elegantly implement complex design patterns. In addition, well-known Python frameworks such as Django, DRF, and SQLAlchemy use metaclasses to provide easy extensibility and simple code reuse.

In this article I’ll tell you why you shouldn’t be afraid to use metaprogramming in your projects and show you what tasks it is best for. You can learn more about metaprogramming capabilities in the Advanced Python course.

For first, let's recall the basics of metaprogramming in Python. It is not superfluous to add that all that is written below refers to the version of Python 3.5 and higher.

A brief excursion into the Python data model

So, we all know that everything in Python is an object, and it is no secret that for each object there is a certain class by which it was generated, for example:

>>> def f(): pass >>> type(f) <class 'function'> 

The type of the object or the class by which the object was spawned can be determined using the built-in type function, which has a rather interesting call signature (it will be discussed a little later). The same effect can be achieved if you display the attribute __class__ from any object.

So, to create functions is a kind of built-in function . Let's see what we can do with it. To do this, take the blank from the built-in types module:

 >>> from types import FunctionType >>> FunctionType <class 'function'> >>> help(FunctionType) class function(object) | function(code, globals[, name[, argdefs[, closure]]]) | | Create a function object from a code object and a dictionary. | The optional name string overrides the name from the code object. | The optional argdefs tuple specifies the default argument values. | The optional closure tuple supplies the bindings for free variables. 

As we can see, any function in Python is an instance of the class described above. Let's now try to create a new function without resorting to its declaration via def . To do this, we need to learn how to create code objects using the built-in compile function:

 #   ,    "Hello, world!" >>> code = compile('print("Hello, world!")', '<repl>', 'eval') >>> code <code object <module> at 0xdeadbeef, file "<repl>", line 1> #  ,     , #      >>> func = FunctionType(code, globals(), 'greetings') >>> func <function <module> at 0xcafefeed> >>> func.__name__ 'greetings' >>> func() Hello, world! 

Fine! With the help of meta-tools we learned how to create functions on the fly, but in practice this knowledge is rarely used. Now let's take a look at how class objects and instance objects of these classes are created:

 >>> class User: pass >>> user = User() >>> type(user) <class '__main__.User'> >>> type(User) <class 'type'> 

Obviously, the User class is used to create an instance of user , it is much more interesting to look at the type class, which is used to create the User class itself. Here we turn to the second variant of calling the built-in type function, which is also a metaclass for any class in Python. A metaclass is by definition a class whose instance is another class. Metaclasses allow us to customize the process of creating a class and partially manage the process of creating an instance of a class.

According to the documentation, the second variant of the type(name, bases, attrs) signature type(name, bases, attrs) returns a new data type or, if simple, a new class, and the name attribute becomes the __name__ attribute of the returned class, bases - the list of parent classes will be available as __bases__ , and attrs , a dict-like object containing all the attributes and methods of a class, will become __dict__ . The principle of operation of the function can be described as a simple pseudocode in Python:

 type(name, bases, attrs) ~ class name(bases): attrs 

Let's see how we can, using only the type call, construct a completely new class:

 >>> User = type('User', (), {}) >>> User <class '__main__.User'> 

As you can see, we do not need to use the class keyword to create a new class, the type function copes without it, now let's consider an example more complicated:

 class User: def __init__(self, name): self.name = name class SuperUser(User): """Encapsulate domain logic to work with super users""" group_name = 'admin' @property def login(self): return f'{self.group_name}/{self.name}'.lower() #     SuperUser "" CustomSuperUser = type( #   'SuperUser', #  ,      (User, ), #         { '__doc__': 'Encapsulate domain logic to work with super users', 'group_name': 'admin', 'login': property(lambda self: f'{self.group_name}/{self.name}'.lower()), } ) assert SuperUser.__doc__ == CustomSuperUser.__doc__ assert SuperUser('Vladimir').login == CustomSuperUser('Vladimir').login 

As you can see from the examples above, the description of classes and functions using the class and def keywords is just syntactic sugar, and any types of objects can be created with ordinary calls to built-in functions. And now, finally, let's talk about how you can use dynamic class creation in real projects.

Dynamic creation of forms and validators

Sometimes we need to validate information from the user or from other external sources according to a previously known data scheme. For example, we want to change the user login form from the admin panel - remove and add fields, change their validation strategy, etc.

To illustrate, let's dynamically create a Django- form, the description of the scheme of which is stored in the following json format:

 { "fist_name": { "type": "str", "max_length": 25 }, "last_name": { "type": "str", "max_length": 30 }, "age": { "type": "int", "min_value": 18, "max_value": 99 } } 

Now, based on the description above, we will create a set of fields and a new form using the type function already known to us:

 import json from django import forms fields_type_map = { 'str': forms.CharField, 'int': forms.IntegerField, } # form_description –  json    deserialized_form_description: dict = json.loads(form_description) form_attrs = {} #            for field_name, field_description in deserialized_form_description.items(): field_class = fields_type_map[field_description.pop('type')] form_attrs[field_name] = field_class(**field_description) user_form_class = type('DynamicForm', (forms.Form, ), form_attrs) >>> form = user_form_class({'age': 101}) >>> form <DynamicForm bound=True, valid=Unknown, fields=(fist_name;last_name;age)> >>> form.is_valid() False >>> form.errors {'fist_name': ['This field is required.'], 'last_name': ['This field is required.'], 'age': ['Ensure this value is less than or equal to 99.']} 

Super! Now you can transfer the created form to a template and render it to the user. The same approach can be used with other frameworks for validation and presentation of data ( DRF Serializers , marshmallow, and others).

Configuring the creation of a new class through the metaclass

Above, we looked at the already “ready” metaclass type , but more often in the code you will create your own metaclasses and use them to configure the creation of new classes and their instances. In general, the “blank” metaclass looks like this:

 class MetaClass(type): """   : mcs –  ,  <__main__.MetaClass> name – ,  ,     ,  "User" bases –   -,  (SomeMixin, AbstractUser) attrs – dict-like ,         cls –  ,  <__main__.User> extra_kwargs –  keyword-     args  kwargs –          """ def __new__(mcs, name, bases, attrs, **extra_kwargs): return super().__new__(mcs, name, bases, attrs) def __init__(cls, name, bases, attrs, **extra_kwargs): super().__init__(cls) @classmethod def __prepare__(mcs, cls, bases, **extra_kwargs): return super().__prepare__(mcs, cls, bases, **kwargs) def __call__(cls, *args, **kwargs): return super().__call__(*args, **kwargs) 

To use this metaclass to configure the User class, use the following syntax:

 class User(metaclass=MetaClass): def __new__(cls, name): return super().__new__(cls) def __init__(self, name): self.name = name 

The most interesting thing is the order in which the Python interpreter invokes the metamethods of the metaclass at the time of the creation of the class itself:

1. The interpreter determines and finds the parent classes for the current class (if any).
2. The interpreter defines the metaclass ( MetaClass in our case).
3. The MetaClass.__prepare__ method is MetaClass.__prepare__ - it must return a dict-like object in which the class attributes and methods will be written. After that, the object will be passed to the MetaClass.__new__ method MetaClass.__new__ through the argument attrs . We will talk about practical use of this method a little later in the examples.
4. The interpreter reads the body of the User class and generates parameters for transferring them to the MetaClass metaclass.
5. The MetaClass.__new__ method is MetaClass.__new__ is a constructor method, returns the created class object. We already met with the arguments name , bases and attrs when we passed them to the type function, and we will talk about the parameter **extra_kwargs little later. If the type of the attrs argument was changed using __prepare__ , then it must be converted to dict before being passed to the super() method call.
6. The MetaClass.__init__ method is MetaClass.__init__ - an initializer method with which you can add additional attributes and methods to a class object. In practice, it is used in cases when metaclasses are inherited from other metaclasses, otherwise everything that can be done in __init__ , it is better to do in __new__ . For example, the __slots__ parameter can be set only in the __new__ method by writing it to the attrs object.
7. At this step, the class is considered created.

Now let's create an instance of our User class and look at the call chain:

 user = User(name='Alyosha') 

1. At the time of the User(...) call, the interpreter calls the MetaClass.__call__(name='Alyosha') method, where it passes the class object and the passed arguments.
2. MetaClass.__call__ calls User.__new__(name='Alyosha') - a constructor method that creates and returns an instance of the User class
3. Next, MetaClass.__call__ calls User.__init__(name='Alyosha') - an initialization method that adds new attributes to the created instance.
4. MetaClass.__call__ returns the created and initialized instance of the User class.
5. At this point, an instance of the class is considered to be created.

This description, of course, does not cover all the nuances of the use of metaclasses, but it is enough to start applying metaprogramming to implement some architectural patterns. Forward to the examples!

Abstract classes

And the very first example can be found in the standard library: ABCMeta - the metaclass allows you to declare any of our classes to be abstract and force all of its heirs to implement predefined methods, properties, and attributes, so take a look:

 from abc import ABCMeta, abstractmethod class BasePlugin(metaclass=ABCMeta): """   supported_formats   run        """ @property @abstractmethod def supported_formats(self) -> list: pass @abstractmethod def run(self, input_data: dict): pass 

If in the successor all abstract methods and attributes are not implemented, then when we try to create an instance of the TypeError class, we will get a TypeError :

 class VideoPlugin(BasePlugin): def run(self): print('Processing video...') plugin = VideoPlugin() # TypeError: Can't instantiate abstract class VideoPlugin # with abstract methods supported_formats 

Using abstract classes helps fix the base class interface immediately and avoid future inheritance errors, such as typos in the name of the overridden method.

Plug-in system with automatic registration

Quite often, metaprogramming is used to implement various design patterns. Almost any known framework uses metaclasses to create registry objects. Such objects store references to other objects and allow them to be quickly received anywhere in the program. Consider a simple example of auto-registration of plug-ins for playing media files of various formats.

Metaclass implementation:

 class RegistryMeta(ABCMeta): """ ,      .     " " -> " " """ _registry_formats = {} def __new__(mcs, name, bases, attrs): cls: 'BasePlugin' = super().__new__(mcs, name, bases, attrs) #     (BasePlugin) if inspect.isabstract(cls): return cls for media_format in cls.supported_formats: if media_format in mcs._registry_formats: raise ValueError(f'Format {media_format} is already registered') #       mcs._registry_formats[media_format] = cls return cls @classmethod def get_plugin(mcs, media_format: str): try: return mcs._registry_formats[media_format] except KeyError: raise RuntimeError(f'Plugin is not defined for {media_format}') @classmethod def show_registry(mcs): from pprint import pprint pprint(mcs._registry_formats) 

And here are the plugins themselves, BasePlugin take the BasePlugin implementation from the previous example:

 class BasePlugin(metaclass=RegistryMeta): ... class VideoPlugin(BasePlugin): supported_formats = ['mpg', 'mov'] def run(self): ... class AudioPlugin(BasePlugin): supported_formats = ['mp3', 'flac'] def run(self): ... 

After the interpreter executes this code, 4 formats and 2 plug-ins that can process these formats will be registered in our registry:

 >>> RegistryMeta.show_registry() {'flac': <class '__main__.AudioPlugin'>, 'mov': <class '__main__.VideoPlugin'>, 'mp3': <class '__main__.AudioPlugin'>, 'mpg': <class '__main__.VideoPlugin'>} >>> plugin_class = RegistryMeta.get_plugin('mov') >>> plugin_class <class '__main__.VideoPlugin'> >>> plugin_class().run() Processing video... 

Here it is worth noting another interesting nuance of working with metaclasses, thanks to the unobvious method resolution order, we can call the show_registry method not only on the RegistyMeta class, but on any other class whose metaclass it is:

 >>> AudioPlugin.get_plugin('avi') # RuntimeError: Plugin is not found for avi 

Using attribute names as metadata

Using metaclasses, you can use class attribute names as metadata for other objects. Nothing is clear? But I’m sure you’ve already seen this approach many times, for example, declarative declaration of model fields in Django

 class Book(models.Model): title = models.Charfield(max_length=250) 

In the example above, the title is the name of the Python identifier, it is also used for the title of the column in the book table, although we have not explicitly indicated this anywhere. Yes, such “magic” can be implemented using metaprogramming. Let's, for example, implement an application error transfer system on the frontend, so that each message has readable code that can be used to translate the message into another language. So, we have a message object that can be converted to json :

 class Message: def __init__(self, text, code=None): self.text = text self.code = code def to_json(self): return json.dumps({'text': self.text, 'code': self.code}) 

All our error messages will be stored in a separate “namespace”:

 class Messages: not_found = Message('Resource not found') bad_request = Message('Request body is invalid') ... >>> Messages.not_found.to_json() {"text": "Resource not found", "code": null} 

Now we want the code become not null , but not_found , for this we write the following metaclass:

 class MetaMessage(type): def __new__(mcs, name, bases, attrs): for attr, value in attrs.items(): #          Message #    code    # ( code   ) if isinstance(value, Message) and value.code is None: value.code = attr return super().__new__(mcs, name, bases, attrs) class Messages(metaclass=MetaMessage): ... 

Let's see how our messages look now:

 >>> Messages.not_found.to_json() {"text": "Resource not found", "code": "not_found"} >>> Messages.bad_request.to_json() {"text": "Request body is invalid", "code": "bad_request"} 

What you need! Now you know what to do so that by the format of the data you can easily find the code that processes them.

Caching class metadata and its heirs

Another frequent case is caching of any static data at the stage of class creation, in order not to waste time on their calculation while the application is running. In addition, some data can be updated when creating new instances of classes, for example, the count of the number of objects created.

How can this be used? Suppose you are developing a framework for building reports and tables, and you have such an object:

 class Row(metaclass=MetaRow): name: str age: int ... def __init__(self, **kwargs): self.counter = None for attr, value in kwargs.items(): setattr(self, attr, value) def __str__(self): out = [self.counter] #  __header__      for name in self.__header__[1:]: out.append(getattr(self, name, 'N/A')) return ' | '.join(map(str, out)) 

We want to save and increase the counter when creating a new series, and also want to generate the header of the resulting table in advance. Metaclass to the rescue!

 class MetaRow(type): #      row_count = 0 def __new__(mcs, name, bases, attrs): cls = super().__new__(mcs, name, bases, attrs) #          cls.__header__ = ['№'] + sorted(attrs['__annotations__'].keys()) return cls def __call__(cls, *args, **kwargs): #      row: 'Row' = super().__call__(*args, **kwargs) #    cls.row_count += 1 #     row.counter = cls.row_count return row 

Here you need to clarify 2 things:

• The Row class has no class attributes with the names name and age — these are type annotations , so they are not in the attrs dictionary keys, and to get the list of fields, we use the class attribute __annotations__ .
• The operation cls.row_count += 1 should have misled you: how is that? After all, cls is a class Row it does not have an attribute row_count . That's right, but as I explained above - if the created class does not have an attribute or method that they are trying to call, then the interpreter goes further along the chain of base classes - if they don’t exist, the metaclass is searched. In such cases, in order not to confuse anyone, it is better to use another record: MetaRow.row_count += 1 .

See how elegantly you can now display the entire table:

 rows = [ Row(name='Valentin', age=25), Row(name='Sergey', age=33), Row(name='Gosha'), ] print(' | '.join(Row.__header__)) for row in rows: print(row) 

 № | age | name 1 | 25 | Valentin 2 | 33 | Sergey 3 | N/A | Gosha 

By the way, the display and work with the table can be encapsulated in any separate class Sheet .

To be continued...

In the next part of this article, I’ll explain how to use metaclasses to debug your application code, how to parameterize the creation of a metaclass, and show the main examples of using the __prepare__ method. Stay tuned!

In more detail about metaclasses and descriptors in Python I will talk in the framework of the Advanced Python intensive.