Geekly Articles each Day
📜 ⬆️ ⬇️

Guide to magical methods in Python

This is a translation of the 1.17 version of the manual from Rafe Kettler.


Content


  1. Introduction
  2. Design and initialization
  3. Redefinition of operators on arbitrary classes
  4. Introducing your classes
  5. Attribute Access Control
  6. Creating arbitrary sequences
  7. Reflection
  8. Called Objects
  9. Context managers
  10. Abstract base classes
  11. Building descriptors
  12. Copying
  13. Using the pickle module on your objects
  14. Conclusion
  15. Appendix 1: How to invoke magic methods.
  16. Appendix 2: Changes in Python 3


Introduction


What are magical methods? They are all in object-oriented Python. These are special methods by which you can add "magic" to your classes. They are always framed by two underscores (for example, __init__ or __lt__ ). Also, they are not as well documented as we would like. All magic methods are described in the documentation, but very randomly and almost without any organization. Therefore, in order to correct what I perceive as a lack of Python documentation, I am going to provide more information about magical methods, written in plain language and richly supplied with examples. I hope you enjoy this guide. Use it as a teaching material, memo, or full description. I just tried to describe magical methods as clearly as possible.
')

Design and initialization.


Everyone knows the most basic magic method, __init__ . With it, we can initialize the object. However, when I write x = SomeClass() , __init__ not the first thing that is called. In fact, an object instance creates a __new__ method, and then the arguments are passed to the initializer. At the other end of the object's life cycle is the __del__ method. Let's take a closer look at these three magic methods:


Connect all together, here is an example of __init__ and __del__ in action:

 from os.path import join class FileObject: '''   ,     ,      .''' def __init__(self, filepath='~', filename='sample.txt'): #   filename  filepath      self.file = open(join(filepath, filename), 'r+') def __del__(self): self.file.close() del self.file


Redefinition of operators on arbitrary classes


One of the great advantages of using magic methods in Python is that they provide an easy way to make objects behave like built-in types. This means that you can avoid the dull, illogical and non-standard behavior of the basic operators. In some languages, it’s common to write something like this:

 if instance.equals(other_instance): # do something

Of course, you can do the same in Python, but this adds confusion and unnecessary verbosity. Different libraries can call the same operations differently, forcing the programmer using them to perform more actions than necessary. Using the power of magic methods, we can determine the desired method ( __eq__ , in this case), and so accurately express what we had in mind :

 if instance == other_instance: #do something

This is one of the strengths of magical methods. The vast majority of them allow us to determine what standard operators will do, so that we can use operators on our classes as if they are built-in types.


Magic methods of comparison


In Python, a lot of magic methods created to determine the intuitive comparison between objects using operators, rather than clumsy methods. In addition, they provide a way to override the default behavior of Python for comparing objects (by reference). Here is a list of these methods and what they do:


For example, consider a class that describes a word. We can compare words lexicographically (alphabetically), which is the default behavior when comparing strings, but we may want to use when comparing some other criterion, such as length or number of syllables. In this example, we will compare in length. Here is the implementation:

 class Word(str): '''  ,     .''' def __new__(cls, word): #    __new__,    str  #       ( ) if ' ' in word: print "Value contains spaces. Truncating to first space." word = word[:word.index(' ')] #  Word       return str.__new__(cls, word) def __gt__(self, other): return len(self) > len(other) def __lt__(self, other): return len(self) < len(other) def __ge__(self, other): return len(self) >= len(other) def __le__(self, other): return len(self) <= len(other)

Now we can create two Word (using Word('foo') and Word('bar') ) and compare them in length. Note that we did not define __eq__ and __ne__ , as this will lead to strange behavior (for example, Word('foo') == Word('bar') will be regarded as true). This does not make sense when testing for equivalence based on length, so we leave the standard equivalence test from str .
Now it seems to be a good time to mention that you do not have to define each of the magic comparison methods in order to fully cover all the comparisons. The standard library kindly provides us with a class-detector in the functools module, which will determine all comparing methods, all you need to do is define only __eq__ and one more ( __gt__ , __lt__ , etc.) This feature is available starting from version 2.7 of Python, but if You are satisfied, you will save a lot of time and effort. To enable it, place @total_ordering above your class definition.


Numerical magic methods


In the same way that you can determine how your objects will be compared by comparison operators, you can define their behavior for numeric operators. Get ready, friends, a lot of them. For better organization, I divided numerical magic methods into 5 categories: unary operators, ordinary arithmetic operators, reflected arithmetic operators (more details later), compound assignments and type conversions.


Unary operators and functions


Unary operators and functions have only one operand - negation, absolute value, and so on.


Common arithmetic operators


Now consider the usual binary operators (and a couple more functions): +, -, * and similar ones. They, for the most part, perfectly describe themselves.


Reflected arithmetic operators


Remember how I said that I was going to dwell on the reflected arithmetic in more detail? You might think that this is some kind of big, scary and incomprehensible concept. In fact, everything is very simple. Here is an example:

 some_object + other

This is the "usual" addition. The only thing that distinguishes the equivalent reflected expression is the order of the terms:

 other + some_object

Thus, all these magic methods do the same as their regular versions, except for performing the operation with the other as the first operand and self as the second. In most cases, the result of the reflected operation is the same as its usual equivalent, so when defining __radd__ you can limit yourself to calling __add__ and that's all. Note that the object to the left of the operator ( other in the example) should not have the usual, unreflected version of this method. In our example, some_object.__radd__ will be called only if __add__ not defined in the other .

Compound assignment


In Python, magic methods for compound assignment are widely represented. You are most likely already familiar with compound assignment; this is a combination of the “ordinary” operator and assignment. If it is still not clear, here is an example:

 x = 5 x += 1 #   x = x + 1

Each of these methods should return the value that will be assigned to the variable on the left (for example, for a += b , __iadd__ should return a + b , which will be assigned to a ). Here is a list:

Magic type conversion methods


In addition, in Python there are many magic methods designed to define behavior for built-in type conversion functions, such as float() . Here they are:


Introducing Your Classes


It is often useful to present a class as a string. There are several methods in Python that you can define to customize the behavior of built-in functions when representing your class.


We are almost done with the boring (and devoid of examples) part of the magic method manual. Now that we have covered the most basic magical methods, it's time to move on to more advanced material.


Attribute Access Control


Many people who come to Python from other languages ​​complain about the absence of real encapsulation for classes (for example, there is no way to define private attributes with public access methods). This is not entirely true: it is just that many things associated with encapsulation are implemented by Python through “magic”, and not by explicit modifiers for methods and fields. See:


You can easily get a problem when defining any attribute that controls access to attributes. Consider an example:

 def __setattr__(self, name, value): self.name = value #  ,    ,     , #  __setattr__(). # ,      self.__setattr__('name', value). #      ,   ,     def __setattr__(self, name, value): self.__dict__[name] = value #      #

Once again, the power of magical methods in Python is incredible, and with great force comes great responsibility. It is important to know how to correctly use magic methods without breaking anything.

So, what have we learned about controlling access to attributes? They do not need to be used lightly. In fact, they tend to be overly powerful and illogical. The reason why they still exist is to satisfy a certain desire: Python is not inclined to prohibit bad things completely, but only to complicate their use. Freedom is paramount, so you can actually do whatever you want. Here is an example of using access control methods (note that we use super, since not all classes have an attribute __dict__):

 class AccessCounter(object): ''',   value      .    ,   value.''' def __init__(self, val): super(AccessCounter, self).__setattr__('counter', 0) super(AccessCounter, self).__setattr__('value', val) def __setattr__(self, name, value): if name == 'value': super(AccessCounter, self).__setattr__('counter', self.counter + 1) #      . #       , #   AttributeError(name) super(AccessCounter, self).__setattr__(name, value) def __delattr__(self, name): if name == 'value': super(AccessCounter, self).__setattr__('counter', self.counter + 1) super(AccessCounter, self).__delattr__(name)]


Creating arbitrary sequences


In Python, there are many ways to make your classes behave like built-in sequences (dictionaries, tuples, lists, strings, and so on). These are, undoubtedly, my favorite magical methods, due to the absurdity of the high degree of control that they give and the magic from which a whole host of global functions suddenly work fine with instances of your classes. But before we get to all sorts of good things, we need to know about the protocols.


Protocols


Now, when it comes to creating your own sequences in Python, it's time to talk about the protocols . Protocols are somewhat similar to interfaces in other languages ​​in that they provide a set of methods that you must implement. However, in Python, the protocols absolutely do not oblige to anything and do not necessarily require to implement any announcement. They are probably more like guidelines.

Why are we talking about protocols? Because the implementation of arbitrary container types in Python entails the use of some of them. First, the protocol for defining immutable containers: to create an immutable container, you only have to define __len__and__getitem__(more about them further). The protocol of the container to be changed requires the same as the container that is not to be changed, plus __setitem__and __delitem__. And finally, if you want your objects to be iterated through iteration, you must determine __iter__which iterator returns. This iterator must comply with an iterator protocol that requires methods __iter__(returns itself) and next.


Magic containers


Without further delay, here are the magic methods used by the containers:


Example


For an example, let's look at a list that implements some functional constructs that you might have found in other languages ​​(Haskell, for example).

 class FunctionalList: '''-       : head, tail, init, last, drop, take.''' def __init__(self, values=None): if values is None: self.values = [] else: self.values = values def __len__(self): return len(self.values) def __getitem__(self, key): #      , list   return self.values[key] def __setitem__(self, key, value): self.values[key] = value def __delitem__(self, key): del self.values[key] def __iter__(self): return iter(self.values) def __reversed__(self): return FunctionalList(reversed(self.values)) def append(self, value): self.values.append(value) def head(self): #    return self.values[0] def tail(self): #      return self.values[1:] def init(self): #      return self.values[:-1] def last(self): #    return self.values[-1] def drop(self, n): #     n return self.values[n:] def take(self, n): #  n  return self.values[:n]

() . , , , ( , ?), Counter , OrderedDict , NamedTuple .



, isinstance() issubclass() , . Here they are:


It may seem that the beneficial uses of these magical methods are few and perhaps this is indeed the case. I do not want to spend too much time on magical methods of reflection, they are not particularly important, but they reflect something important about object-oriented programming in Python and Python in general: there is almost always an easy way to do something, even if needed this “something” occurs very rarely. These magic methods may not look useful, but if you ever need them, you will be happy to remember that they exist (and for this you are reading this guide!).


Called Objects


As you probably already know, in Python, functions are first class objects. This means that they can be passed to functions or methods just like any other objects. This is an incredibly powerful feature.

The special magic method allows instances of your class to behave as if they were functions, that is, you can "call" them, pass them into functions that accept functions as arguments, and so on. This is another handy feature that makes programming in Python so enjoyable.


__call__In particular, it can be useful in classes whose instances often change their state. Invoking an instance can be an intuitive and elegant way to change the state of an object. An example would be a class representing the position of an object on a plane:

 class Entity: ''',    . "",    .''' def __init__(self, size, x, y): self.x, self.y = x, y self.size = size def __call__(self, x, y): '''  .''' self.x, self.y = x, y # ...


Context managers


In Python 2.5, a new keyword was introduced along with a new way to reuse code, a keyword with. The concept of context managers was not new to Python (it was implemented earlier as part of a library), but in PEP 343 it reached the status of a language construct. You could already see expressions with with:

 with open('foo.txt') as bar: #  -   bar

Context managers allow you to perform some actions for customization or cleanup when object creation is wrapped in an operator with. The behavior of the context manager is determined by two magic methods:


__enter__and __exit__can be useful for specific classes with well-described and common behavior for setting them up and clearing resources. You can use these methods to create common context managers for different objects. Here is an example:

 class Closer: '''        close  with-.''' def __init__(self, obj): self.obj = obj def __enter__(self): return self.obj #     with- def __exit__(self, exception_type, exception_val, trace): try: self.obj.close() except AttributeError: #     close print 'Not closable.' return True #

An example of using Closerwith an FTP connection (a socket that has a close method):

 >>> from magicmethods import Closer >>> from ftplib import FTP >>> with Closer(FTP('ftp.somesite.com')) as conn: ... conn.dir() ... # output omitted for brevity >>> conn.dir() # long AttributeError message, can't use a connection that's closed >>> with Closer(int(5)) as i: ... i += 1 ... Not closable. >>> i 6

See how our wrapper elegantly handles both the right and the wrong objects. This is the power of context managers and magic methods. Notice that the standard Python library includes a contextlib module , which includes contextlib.closing()a context manager, which does roughly the same thing (without any handling of the case where the object has no method close()).


Abstract base classes


See http://docs.python.org/2/library/abc.html .


Building descriptors


Descriptors are such classes, with the help of which you can add your logic to access events (receiving, modifying, deleting) to attributes of other objects. Descriptors are not intended to be used by themselves; rather, it is assumed that they will be owned by any related classes. Descriptors can be useful for building object-oriented databases or classes whose attributes are dependent on each other. In particular, descriptors are useful when representing attributes in several calculus systems or any calculated attributes (such as the distance from the starting point to the point represented by the attribute on the grid).

For a class to become a handle, it must implement at least one method from __get__, __set__or __delete__. Let's look at these magical methods:


Now an example of the useful use of descriptors: unit conversion.

 class Meter(object): '''  .''' def __init__(self, value=0.0): self.value = float(value) def __get__(self, instance, owner): return self.value def __set__(self, instance, value): self.value = float(value) class Foot(object): '''  .''' def __get__(self, instance, owner): return instance.meter * 3.2808 def __set__(self, instance, value): instance.meter = float(value) / 3.2808 class Distance(object): ''',  ,       .''' meter = Meter() foot = Foot()


Copying


In Python, the assignment operator does not copy objects, but only adds another reference. But for collections that contain mutable elements, sometimes you need a full copy so that you can change the elements of one sequence without affecting the other. Here comes into play copy. Fortunately, modules in Python do not have a reason, so we can not worry that they suddenly begin to copy themselves uncontrollably and soon Linux robots will flood the entire planet, but we must tell Python how to copy correctly.


When to use these magical methods? As usual - in any case, when you need more than the standard behavior. For example, if you are trying to copy an object that contains a cache as a dictionary (perhaps a very large dictionary), then you may not need to copy the entire cache, but do just one in the shared memory of objects.


Using the pickle module on your objects


Pickle is a module for serializing Python data structures and can be incredibly useful when you need to save the state of an object and restore it later (usually for caching purposes). In addition, it is also an excellent source of experiences and confusion.

Serialization is so important that in addition to its module ( pickle) it has its own protocol and its own magic methods. But for a start on how to serialize pickle with already existing data types (quietly skip if you already know).


Briefly about serialization


Let's dive into serialization. Suppose you have a dictionary that you want to save and restore later. You must write its contents to a file, carefully making sure that you are writing with the correct syntax, then restoring it, either by running exec()or reading the file. But this is at best risky: if you store important data in the text, it can be damaged or changed in many ways, in order to crash your program or, in general, run some dangerous code on your computer. Better use pickle:

 import pickle data = {'foo': [1, 2, 3], 'bar': ('Hello', 'world!'), 'baz': True} jar = open('data.pkl', 'wb') pickle.dump(data, jar) #     jar jar.close()

And now, after a few hours, we need our dictionary again:

 import pickle pkl_file = open('data.pkl', 'rb') #  data = pickle.load(pkl_file) #    print data pkl_file.close()

What happened?Exactly what was expected. dataas if it had always been there.

Now, a bit of caution: pickle is not perfect. Its files are easily spoiled accidentally or intentionally. Pickle may be safer than text files, but it can still be used to run malicious code. In addition, it is incompatible between different versions of Python, so if you will distribute your objects using pickle, do not expect that all people will be able to use them. However, a module can be a powerful tool for caching and other common serialization tasks.


Serialize your own objects.


The pickle module is not only for built-in types. It can be used with every class that implements its protocol. This protocol contains four optional methods that allow you to configure how pickle will handle them (there are some differences for C extensions, but this is beyond the scope of our tutorial):


Example


For example, we will describe a slate ( Slate), which remembers what and when it was written on it. However, specifically this board becomes clean every time it is serialized: the current value is not saved.

 import time class Slate: ''',     .      .''' def __init__(self, value): self.value = value self.last_change = time.asctime() self.history = {} def change(self, new_value): #  .     . self.history[self.last_change] = self.value self.value = new_value self.last_change = time.asctime() def print_changes(self): print 'Changelog for Slate object:' for k, v in self.history.items(): print '%s\t %s' % (k, v) def __getstate__(self): #    self.value or self.last_change. #   " "  . return self.history def __setstate__(self, state): self.history = state self.value, self.last_change = None, None


Conclusion


The goal of this guide is to bring something to everyone who reads it, regardless of its experience in Python or object-oriented programming. If you are new to Python, you have gained valuable knowledge of the basics of writing functional, elegant and easy-to-use classes. If you are a mid-level programmer, then you may have found some nice new ideas and strategies, some good ways to reduce the amount of code written by you and your clients. If you are a Pythonist expert, then you have updated some of your, perhaps, forgotten knowledge, and maybe you have found a couple of new tricks. Regardless of your level, I hope that this journey through special Python methods was truly magical (I could not resist).


Addition 1: How to cause magic methods


Some of the magic methods are directly related to the built-in functions; in this case, it’s quite obvious how to call them. However, this is not always the case. This addition is dedicated to uncovering unobvious syntax leading to invocation of magic methods.
Magic methodWhen it is called (example)Explanation
__new__(cls [,...])instance = MyClass(arg1, arg2)__new__ called when creating an instance
__init__(self [,...])instance = MyClass(arg1, arg2)__init__ called when creating an instance
__cmp__(self, other)self == other, self > other, Etc.Called for any comparison.
__pos__(self)+selfUnary plus sign
__neg__(self)-selfUnary minus sign
__invert__(self)~selfBitwise inversion
__index__(self)x[self]Transformation when an object is used as an index
__nonzero__(self)bool(self) , if self:Object Boolean
__getattr__(self, name)self.name # nameTrying to get a non-existent attribute.
__setattr__(self, name, val)self.name = valAssigning to any attribute
__delattr__(self, name)del self.nameAttribute removal
__getattribute__(self, name)self.nameGet any attribute
__getitem__(self, key)self[key]Getting an item through the index
__setitem__(self, key, val)self[key] = valAssignment to an element through an index
__delitem__(self, key)del self[key]Deleting an item by index
__iter__(self)for x in selfIteration
__contains__(self, value)value in self , value not in selfAffiliation checking with in
__call__(self [,...])self(args)"Call" instance
__enter__(self)with self as x:with
__exit__(self, exc, val, trace)with self as x:with
__getstate__(self)pickle.dump(pkl_file, self)Serialization
__setstate__(self)data = pickle.load(pkl_file)Serialization


, , .


2: 3


, 3 2.x :



Source: https://habr.com/ru/post/186608/

More articles:


All Articles