Using metaclasses in Python

Some metaprogramming tools are not often used daily.
work, as usual in OOP classes or the same decorators. For understanding the same goals
introducing such tools into the language requires specific examples of industrial
applications, some of which are listed below.

Introduction to Metaclasses

')
So, classical OOP implies the presence of only classes and objects.
Class - template for the object; when declaring a class, all the mechanics are indicated
the work of each particular "incarnation": set the data encapsulated
in the object, and methods for working with this data.


Python expands the classical paradigm, and the classes themselves in it also become
peer objects that can be changed, assigned to a variable and
pass in functions. But if a class is an object, which class does it correspond to?
By default, this class (metaclass) is called type.

You can inherit from metaclass by getting a new metaclass, which, in its
queue can be used when defining new classes. In this way,
a new “dimension” of inheritance appears, adding to the inheritance hierarchy
classes: metaclass -> class -> object.

Simple example

Suppose we are tired of setting the attributes in the constructor __init __ (self, * args,
** kwargs) I would like to speed up this process so that
the ability to set attributes directly when creating a class object. With the usual
class this will not work:

   >>> class Man (object):
   >>> pass
   >>> me = Man (height = 180, weight = 80)
   Traceback (most recent call last):
   File "<stdin>", line 20, in <module>
       TypeError: object .__ new __ () takes no parameters

The object is constructed by calling the class with the "()" operator. Create inheritance from
type metaclass to override this statement:

   >>> class AttributeInitType (type):
   >>> def __call __ (self, * args, ** kwargs):
   >>> "" "Calling a class creates a new object." ""
   >>> # First of all we create the object itself ...
   >>> obj = type .__ call __ (self, * args)
   >>> # ... and add the arguments passed to it as attributes.
   >>> for name in kwargs:
   >>> setattr (obj, name, kwargs [name])
   >>> # return the finished object
   >>> return obj

Now create a class using the new metaclass:

   >>> class Man (object):
   >>> __metaclass__ = AttributeInitType

Voila:

   >>> me = Man (height = 180, weigth = 80)
   >>> print me.height
   180

Language extension (abstract classes)

The Python core is relatively small and simple, a set of built-in tools.
small, allowing developers to quickly master the language.


However, programmers involved in creating, for example, frameworks and
related special sublanguages ​​(domain specific languages) are provided
quite flexible tools.

Abstract classes (or their slightly different form - interfaces) - common
and popular among programmers method of determining the interface part
class. Usually such concepts are laid in the core of the language (as in Java or C ++),
Python also allows you to gracefully and easily implement their own means, in
Particulars - with the help of metaclasses and decorators.

Consider the work of the abc library from the implementation proposal for the standard library.

abc

Using asbestos classes is very easy. Create an abstract base class
with a virtual method and try to create a class-successor without defining this method:

 >>> from abc import ABCMeta, abstractmethod
 >>> class A (object):
 >>> __metaclass __ = ABCMeta
 >>> @abstractmethod
 >>> def foo (self): pass
 >>> 
 >>> A ()  
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 TypeError: Can't instantiate

Did not work. Now we define the desired method:

  
 >>> class C (A):
 >>> def foo (self): print (42)
 >>> C
 <class '__main __. C'>
 >>> a = C ()
 >>> a.foo ()
 42

We learn how this is implemented in the metaclass (omitting some other possibilities
module abc) ABCMeta:

 >>> class ABCMeta (type):
 >>> def __new __ (mcls, name, bases, namespace):
 >>> bases = _fix_bases (bases)
 >>> cls = super (ABCMeta, mcls) .__ new __ (mcls, name, bases, namespace)
 >>> # Find the set (set) of abstract method names among our own
 >>> # methods and methods of ancestors
 >>> abstracts = set (name
 >>> for name, value in namespace.items ()
 >>> if getattr (value, "__isabstractmethod__", False))
 >>> for base in bases:
 >>> for name in getattr (base, "__abstractmethods__", set ()):
 >>> value = getattr (cls, name, None)
 >>> if getattr (value, "__isabstractmethod__", False):
 >>> abstracts.add (name)
 >>> cls .__ abstractmethods__ = frozenset (abstracts)
 >>> return cls

The _fix_bases method adds the hidden class _Abstract to the number of ancestors
abstract class. _Abstract itself checks if anything is left
set (set) __abstractmethods__; if left, throws an exception.

 >>> class _Abstract (object):
 >>> def __new __ (cls, * args, ** kwds):
 >>> am = cls .__ dict __. get ("__ abstractmethods__")
 >>> if am:
 >>> raise TypeError ("can't instantiate abstract class% s"
 >>> "with abstract methods% s"%
 >>> (cls .__ name__, "," .join (sorted (am))))
 >>> return super (_Abstract, cls) .__ new __ (cls, * args, ** kwds)
 >>>
 >>> def _fix_bases (bases):
 >>> for base in bases:
 >>> if issubclass (base, _Abstract):
 >>> # _Abstract is already among ancestors
 >>> return bases
 >>> if object in bases:
 >>> # Replace object with _Abstract if the class is directly inherited from object
 >>> # and not listed among other ancestors
 >>> return tuple ([_ Abstract if base is object else base
 >>> for base in bases])
 >>> # Add _Abstract to the end otherwise
 >>> return bases + (_Abstract,)

In each abstract class is stored on the "frozen" set (frozenset)
abstract methods; that is, those methods (object functions) that have
the __isabstractmethod__ attribute exposed by the corresponding decorator:

 >>> def abstractmethod (funcobj):
 >>> funcobj .__ isabstractmethod__ = True
 >>> return funcobj

So, the abstract method gets the __isabstractmethod__ attribute when it is assigned
decorator. Attributes after inheriting from an abstract class are collected in
set "__abstractmethods__" of a heir class. If the set is not empty, and
the programmer tries to create a class object, then an exception will be thrown
TypeError with a list of undefined methods.

Conclusion

Simply? Simply. Is the language expanded? Expanded Comments, as they say, are superfluous.

DSL in Django

One of the advanced examples of DSL is the Django ORM mechanism using the Model class as an example.
metaclass modelbase. Specifically, the connection with the database is not interesting here, it has
It makes sense to concentrate on creating an instance of a class derived from Model.

Most of the following subsection is a detailed code review.
ModelBase. Readers who do not need details, just read the output
at the end of the "Django" section.

Analysis of ModelBase metaclass

All the mechanics of the ModelBase metaclass are concentrated in place.
overriding the __new__ method called immediately before creation
model class instance:

   >>> class ModelBase (type):
   >>> "" "
   >>> Metaclass for all models.
   >>> "" "
   >>> def __new __ (cls, name, bases, attrs):
   >>> super_new = super (ModelBase, cls) .__ new__
   >>> parents = [b for b in bases if isinstance (b, ModelBase)]
   >>> if not parents:
   >>> # If this isn't a subclass of Model, don't do anything special.
   >>> return super_new (cls, name, bases, attrs)

At the very beginning of the method an instance of the class is simply created and, if this class is not
inherits from Model, just returns.

All specific model class options are collected in the _meta class attribute, which
can be created from scratch, inherited from an ancestor, or adjusted to
Local class Meta:

   >>> #Class creation
   >>> module = attrs.pop ('__ module__')
   >>> new_class = super_new (cls, name, bases, {'__module__': module})
   >>> attr_meta = attrs.pop ('Meta', None)
   >>> abstract = getattr (attr_meta, 'abstract', False)
   >>> if not attr_meta:
   >>> meta = getattr (new_class, 'Meta', None)
   >>> else:
   >>> meta = attr_meta
   >>> base_meta = getattr (new_class, '_meta', None)

In addition, we see that a class can be abstract, not relevant
any table in the database.

The moment of truth in the process of creating a class of model comes with the introduction into it
default settings:

   >>> new_class.add_to_class ('_ meta', Options (meta, ** kwargs))

add_to_class either calls the argument’s method contribute_to_class or, if
there is none, just adds the named attribute to the class.

The Options class in its contribute_to_class makes the _meta attribute a reference to
himself and collects in it various parameters, like the name of the base table
data, model field list, model virtual field list, access rights, and
others. He also carries out tests of connections with other models for uniqueness.
field names in the database.

Next, in the __new__ method, the named class is added to the non-abstract class.
exceptions:

   >>> if not abstract:
   >>> new_class.add_to_class ('DoesNotExist',
   >>> subclass_exception ('DoesNotExist', ObjectDoesNotExist, module))
   >>> new_class.add_to_class ('MultipleObjectsReturned',
   >>> subclass_exception ('MultipleObjectsReturned',
   >>> MultipleObjectsReturned, module))

If the parent class is not abstract, and the parameters are not set explicitly in the local
class Meta, then inheriting ordering and get_latest_by parameters:

   >>> if base_meta and not base_meta.abstract: >>> if not hasattr (meta, 'ordering'): >>> new_class._meta.ordering = base_meta.ordering >>> if not hasattr (meta, 'get_latest_by') : >>> new_class._meta.get_latest_by = base_meta.get_latest_by 

The default manager should be zero. If such a model already exists, terminate processing by returning this model:

   >>> if getattr (new_class, '_default_manager', None):
   >>> new_class._default_manager = None
   >>>        
   >>> m = get_model (new_class._meta.app_label, name, False)
   >>> if m is not None:
   >>> return m

Nothing special, just adding attributes to the model class with which it was
created by:

   >>> for obj_name, obj in attrs.items ():
   >>> new_class.add_to_class (obj_name, obj)

Now you need to go through the fields of the model and find one-to-one relationships,
which will be used below:

   >>> # Do the appropriate setup for any model parents.
   >>> o2o_map = dict ([(f.rel.to, f) for f in new_class._meta.local_fields
   >>> if isinstance (f, OneToOneField)])

Passing through the ancestors of the model for the inheritance of various fields, with discarding those
that are not heirs to the Model. Further comments are translated, which
enough to understand what is happening:

   >>> for base in parents:
   >>> if not hasattr (base, '_meta'):
   >>> # Models without _meta are not active and do not represent interest
   >>> continue
   >>>
   >>> # All fields of any type for this model
   >>> new_fields = new_class._meta.local_fields + \
   >>> new_class._meta.local_many_to_many + \
   >>> new_class._meta.virtual_fields
   >>> field_names = set ([f.name for f in new_fields])
   >>>
   >>> if not base._meta.abstract:
   >>> # Processing "concrete" classes ...
   >>> if base in o2o_map:
   >>> field = o2o_map [base]
   >>> field.primary_key = True
   >>> new_class._meta.setup_pk (field)
   >>> else:
   >>> attr_name = '% s_ptr'% base._meta.module_name
   >>> field = OneToOneField (base, name = attr_name,
   >>> auto_created = True, parent_link = True)
   >>> new_class.add_to_class (attr_name, field)
   >>> new_class._meta.parents [base] = field
   >>>
   >>> else:
   >>> # .. and abstract.
   >>>
   >>> # Check for name collisions between classes
   >>> # declared in this class and in the abstract ancestor
   >>> parent_fields = base._meta.local_fields + base._meta.local_many_to_many
   >>> for field in parent_fields:
   >>> if field.name in field_names:
   >>> raise FieldError ('Local field% r in class% r clashes' \
   >>> 'with field of similar name from' \
   >>> 'abstract base class% r'% \
   >>> (field.name, name, base .__ name__))
   >>> new_class.add_to_class (field.name, copy.deepcopy (field))
   >>>
   >>> # All non-abstract parents are transferred to the heir.
   >>> new_class._meta.parents.update (base._meta.parents)
   >>>
   >>> # Basic Managers inherit from abstract classes
   >>> base_managers = base._meta.abstract_managers
   >>> base_managers.sort ()
   >>> for _, mgr_name, manager in base_managers:
   >>> val = getattr (new_class, mgr_name, None)
   >>> if not val or val is manager:
   >>> new_manager = manager._copy_to_model (new_class)
   >>> new_class.add_to_class (mgr_name, new_manager)
   >>>
   >>> # Virtual fields (like GenericForeignKey) are taken from the parent
   >>> for field in base._meta.virtual_fields:
   >>> if base._meta.abstract and field.name in field_names:
   >>> raise FieldError ('Local field% r in class% r clashes' \
   >>> 'with field of similar name from' \
   >>> 'abstract base class% r'% \
   >>> (field.name, name, base .__ name__))
   >>> new_class.add_to_class (field.name, copy.deepcopy (field))
   >>>

Abstract classes of models are not registered anywhere:

   >>> if abstract:
   >>> # Abstract models cannot be instantiated and do not appear
   >>> # in the list of models for the application, therefore, they look a little different than
   >>> # normal models
   >>> attr_meta.abstract = False
   >>> new_class.Meta = attr_meta
   >>> return new_class

Normal ones are registered and returned from the list of registered ones.
classes of models:

   >>> new_class._prepare ()
   >>> register_models (new_class._meta.app_label, new_class)
   >>> return get_model (new_class._meta.app_label, name, False)

Conclusion

So let's summarize. Why did you need metaclasses?

1) The model class must have a set of required parameters (table name, name
jango applications, list of fields, links to other models and many others) in
the _meta attribute, which are defined when creating each class that inherits
from Model.

2) These parameters are inherited in a complex way from ordinary and abstract
ancestral classes, which is ugly to lay in the class itself.

3) It is possible to hide what is happening from the programmer using
framework

Zamadanitsa

1) If you do not explicitly specify the class inheritance from object, then the class uses
the metaclass specified in the global variable __metaclass__, which sometimes can
to be comfortable with repeated use of your own metaclass within
single module. A simple example given at the beginning of a note can be redone.
in the following way:

   class AttributeInitType (type):
       def __call __ (self, * args, ** kwargs):
       obj = type .__ call __ (self, * args)
       for name in kwargs:
       setattr (obj, name, kwargs [name])
       return obj

   __metaclass__ = AttributeInitType

   class Man:
       pass

   me = Man (height = 180, weigth = 80)
   print me.height

   The standard stream will display:
   180

2) There is a Python superguru, Tim Peters. He very well said about
the use of metaclasses and similar tools from the category of black magic of Python:

     Metaclasses are more than 99% of users should ever worry
     about.  If you
     people who need it
     they do not need an explanation of why).

In Russian it sounds like this:

     Metaclasses are too much for most users.  If at all you are wondering
     the question is whether they are needed, they are definitely not needed.  They are used only by people
     who know exactly what they are doing and do not need explanations.

The moral here is simple: do not be wise. Metaclasses in most cases - too much. The pythonist should be guided by the principle of least surprise;
change the classic scheme of the PLO is not just for the sake of narcissism.

References based on

English Wikipedia - a simple example is borrowed from here
PEP-3119 is here
abstract classes are described in their full version.
Roller
in English , detailed talk about metaclasses in Python with examples
of use. There you can find links to the article itself with examples, very
instructive.