Notes on the Python object system part 3

The third part of the notes about the python object system (the first and second parts). The article tells why c .__ call __ () is not the same as c (), how to implement singleton using metaclasses, what name mangling is and how it works.

c .__ call__ vs c (), c .__ setattr__ vs setattr

It is easy to verify that x (arg1, arg2) is not equivalent to x .__ call __ (arg1, arg2) for new classes, although this is true for old classes.
')
>>> class C ( object ):
... pass
...
>>> c = C()
>>> c . __call__ = lambda : 42
>>> c()
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
TypeError : 'C' object is not callable
>>> C . __call__ = lambda self : 42
>>> c()
42

Actually correct:

c () <=> type © . __call __ (s)

Absolutely the same situation with __setattr __ / setattr and many other magic (and special) methods and corresponding built-in functions that are defined for all objects, including for objects of type - classes.

Why this was done can be seen on the example of setattr [1].
First , make sure that setattr (a, 'x' , 1 ) <==> type (a) . __setattr __ (a, 'x' , 1 ).

a . x = 1 <=> setattr (a, 'x' , 1 )

>>> class A ( object ): pass
...
>>> a = A()
>>> a . x = 1
>>> a
<__main__.A object at 0x7fafa9b26f90>
>>> setattr (a, 'y' , 2 )
>>> a . __dict__
{'y': 2, 'x': 1}

Using the __setattr__ method, we set a new attribute that will go to __dict__

>>> a . __setattr__( 'z' , 3 )

everything seems to be correct:

>>> a . __dict__
{'y': 2, 'x': 1, 'z': 3}

But:

Set a deliberately wrong method in a .__ setattr__:

>>> a . __setattr__ = lambda self : 42

The call that leads to the error:

>>> a . __setattr__( 'z' , 4 )
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
TypeError : <lambda>() takes exactly 1 argument (2 given)

However, despite this, setattr works:

>>> setattr (a, 'foo' , 'bar' )
>>> a . __dict__
{'y': 2, 'x': 1, '__setattr__': <function <lambda> at 0x7fafa9b3a140>, 'z': 3, 'foo': 'bar'}

But if you override the class method:

>>> A . __setattr__ = lambda self : 42

then setattr for the class instance will generate an error:

>>> setattr (a, 'baz' , 'quux' )
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
TypeError : <lambda>() takes exactly 1 argument (3 given)

Why was this done?
Let setattr (a, 'x', 1) be the same as a .__ setattr __ ('x', 1), then

>>> class A ( object ):
... def __setattr__ ( self , attr, value):
... print 'for instances' , attr, value
... object . __setattr__( self , attr, value)
...
>>> a = A()

Set a new attribute for aax = 1 <==> a .__ setattr __ ('x', 1)
Everything is normal:

>>> a . __setattr__( 'x' , 1 )
for instances x 1
>>> a . __dict__
{'x': 1}

Now let's try to set a new attribute for the class itself, it’s also an object: A.foo = 'bar' <==> A .__ setattr __ ('foo', 'bar')

>>> A . __setattr__( 'foo' , 'bar' )
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
TypeError : unbound method __setattr__() must be called with A instance as first argument (got str instance instead)

Everything is logical, according to the attribute search algorithm in classes (types), first the attribute is searched in __dict__ class (type):

>>> A . __dict__[ '__setattr__' ]
<function __setattr__ at 0x7f699d22fa28>

But the fact is that it is intended for instances of a class, and not for the class itself. Therefore, calling A .__ setattr __ ('foo', 'bar') will be incorrect. And that is why setattr () should do an explicit search in the class (type) of the object. Actually, for the same reason, this was done for other magic methods __add__, __len__, __getattr__, etc.

Class, as callable type

The class (type) is the callable type, and its call is the object constructor.

>>> class C ( object ):
... pass
...
>>> ()
<__main__.C object at 0x1121e10>

Equivalent to:

>>> type (C) . __call__(C)
<__main__.C object at 0x1121ed0>

Because C is a normal class, then its metaclass is type, so the call type (C) .__ call __ (C) <==> type .__ call __ (C) will be used. Inside type .__ call __ (C), a call to C .__ new __ (cls, ...) and C .__ init __ (self, ...) is already in progress.

The important thing is that both __new__ and __init__ are searched for using the usual attribute search algorithm in the class. And in the absence of them in C .__ dict__, methods from the parent class object: object .__ new__ and object .__ init__ will be called, while the __call__ method is a method of the class (type) of the object - type: type .__ call __ (C).

Singleton v.2

Knowing this, we will create a metaclass implementation of a singleton.

What do we need from singleton? To call A () return the same object.

A () <=> type (A) .__ call __ (A)

So, we need to change the behavior of the __call__ method, which is defined in the metaclass. We will do this, without forgetting that, in general, in __call__, any parameters can be transmitted.

>>> class SingletonMeta ( type ):
... def __call__ (cls, * args, ** kw):
... return super (SingletonMeta, cls) . __call__( * args, ** kw)
...
>>>

The cap is ready.
Let the single object be stored in the instance class attribute. To do this, initialize in cls.instance in __init__.

>>> class SingletonMeta ( type ):
... def __init__ (cls, * args, ** kw):
... cls . instance = None
... def __call__ (cls, * args, ** kw):
... return super (SingletonMeta, cls) . __call__( * args, ** kw)
...
>>>


__call__:

>>> class SingletonMeta ( type ):
... def __init__ (cls, * args, ** kw):
... cls . instance = None
... def __call__ (cls, * args, ** kw):
... if cls . instance is None :
... cls . instance = super (SingletonMeta, cls) . __call__( * args, ** kw)
... return cls . instance
...
>>> class C ( object ):
... __metaclass__ = SingletonMeta
...

We check that everything works as it should.

>>> C() is C()
True
>>> a = C()
>>> b = C()
>>> a . x = 42
>>> b . x
42
>>>

Callable type as a metaclass

A metaclass can be not only an object of type type, but generally any callable type.

It is enough just to create a function in which a class is created using the type metaclass.

>>> def mymeta (name, bases, attrs):
... attrs[ 'foo' ] = 'bar'
... return type (name, bases, attrs)
...
>>> class D ( object ):
... __metaclass__ = mymeta
...
>>> D()
<__main__.D object at 0x7fafa9abc090>
>>> d = D()
>>> d . foo
'bar'
>>> d . __dict__
{}
>>> D . __dict__
<dictproxy object at 0x7fafa9b297f8>
>>> dict (D . __dict__)
{'__module__': '__main__', '__metaclass__': <function mymeta at 0x7fafa9b3a9b0>, '__dict__': <attribute '__dict__' of 'D' objects>, 'foo': 'bar', '__weakref__': <attribute '__weakref__' of 'D' objects>, '__doc__': None}

Class definitions

The construction statement of a class definition is simply a construction. Like any statement, it can appear anywhere in the program code.

>>> if True :
... class A ( object ):
... def foo ( self ):
... print 42
...
>>> A
<class '__main__.A'>
>>> A() . foo()
42
>>>

In the 'class' construction, any defined “inside” variables, functions, classes are accumulated in __dict__. And in the definition you can use any other constructions - cycles, if'y :.

Therefore, you can do this:

>>> class A ( object ):
... if 1 > 2 :
... def foo ( self ):
... print '1>2'
... else :
... def bar ( self ):
... print 'else'
...
>>>
>>> A()
<__main__.A object at 0x7fafa9abc150>
>>> A() . foo()
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
AttributeError : 'A' object has no attribute 'foo'
>>> A() . bar()
else

or so
>>> class A ( object ):
... if 1 > 2 :
... x = 1
... def foo ( self ):
... print 'if'
... else :
... y = 1
... def bar ( self ):
... print 'else'
...
>>> A . x
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
AttributeError : type object 'A' has no attribute 'x'
>>> A . y
1
>>> A . foo
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
AttributeError : type object 'A' has no attribute 'foo'
>>> A . bar
<unbound method A.bar>
>>> A . bar()
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
TypeError : unbound method bar() must be called with A instance as first argument (got nothing instead)
>>> A() . bar()
else
>>>

You can put one definition into another.

>>> class A ( object ):
... class B ( object ):
... pass
...
...
>>> A()
<__main__.A object at 0x7fafa9abc2d0>
>>> A . __dict__
<dictproxy object at 0x7fafa9b340f8>
>>> dict (A . __dict__)
{'__dict__': <attribute '__dict__' of 'A' objects>, '__module__': '__main__', 'B': <class '__main__.B'>, '__weakref__': <attribute '__weakref__' of 'A' objects>, '__doc__': None}
>>> A . B()
<__main__.B object at 0x7fafa9abc310>


Or dynamically create class methods:

>>> FIELDS = [ 'a' , 'b' , 'c' ]
>>> class A ( object ):
... for f in FIELDS:
... locals ()[f] = lambda self : 42
...
>>> a = A()
>>> a . a()
42
>>> a . b()
42
>>> a . c()
42
>>> a . d()
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
AttributeError : 'A' object has no attribute 'd'
>>>

Although of course this is certainly not recommended to do in normal practice, and it is better to use more idiomatic means.

Name mangling

And about class definitions. About name mangling.

Any attribute inside a classname class definition of the form ".__ {attr}" (attr has no more than one _ at the end) is replaced by "_ {classname} __ {attr}". Thus, inside classes you can have “hidden” private attributes that are not “visible” to the heirs and instances of the class.

>>> class A ( object ):
... __private_foo =1
...
>>> A . __private_foo
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
AttributeError : type object 'A' has no attribute '__private_foo'

You can see the variable like this:

>>> A . _A__private_foo
1

Well, it is stored in __dict__ class:

>>> dict (A . __dict__)
{'__dict__': <attribute '__dict__' of 'A' objects>, '_A__private_foo': 1, '__module__': '__main__', '__weakref__': <attribute '__weakref__' of 'A' objects>, '__doc__': None}
>>>

Access heirs do not have:

>>> class B (A):
... def foo ( self ):
... print self . __private_foo
...
>>> B() . foo()
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
File "<stdin>" , line 3 , in foo
AttributeError : 'B' object has no attribute '_B__private_foo'

In principle, to provide access external access to the attributes of the type __ {attr} inside the class definition, i.e. bypass name_mangling, you can use __dict__.

>>> class C ( object ):
... def __init__ ( self ):
... self . __dict__[ '__value' ] = 1
...
>>> C() . __value
1
>>>

However, such things are highly discouraged due to the fact that access to such attributes will not be possible within the definition of any other class due to the substitution of ".__ {attr}" for "._ {classname} __ {attr}" regardless which object or class they belong to, i.e.

>>> class D ( object ):
... def __init__ ( self ):
... self . c = C() . __value
...
>>> D()
Traceback (most recent call last):
File "<stdin>" , line 1 , in <module>
File "<stdin>" , line 3 , in __init__
AttributeError : 'C' object has no attribute '_D__value'
>>> C() . __value
1
>>>

Although C () .__ value will work fine outside the class definition. To get around also have to use __dict __ ['__ value'].

Links

• Unifying types and classes in Python is the main document explaining what, how and why in new classes.
• Making Types Look More Like Classes - PEP 252, describing the difference between the old classes and the new ones.
• Built-in functions - a detailed description of the work of all built-in functions.
• Data model - a detailed description of the python data model.
• Python types and objects - an explanation of the python object model on simple examples with pictures.

Notes

[1] The official documentation gives an example of __len __ / len and __hash __ / hash.