Python: decorating decorators. Again

Last year on Habré there was already a very detailed article in two parts about decorators. The purpose of this new article is to cut to the chase and immediately take up interesting, meaningful examples in order to have time to figure out the examples that are even more tricky than in previous articles.
The target audience is programmers, already familiar (for example, in C #) with higher-order functions and with closures, but accustomed to the fact that annotations for functions are “meta-information”, which manifests itself only in reflection. A special feature of Python, immediately apparent to such programmers, is that the presence of a decorator before declaring a function allows changing the behavior of this function:



How it works? Nothing tricky: the decorator is just a function that takes an argument to decorate a function and returns a “fixed” one:
')

def timed(fn): def decorated(*x): start = time() result = fn(*x) print "Executing %s took %d ms" % (fn.__name__, (time()-start)*1000) return result return decorated @timed def cpuload(): load = psutil.cpu_percent() print "cpuload() returns %d" % load return load print "cpuload.__name__==" + cpuload.__name__ print "CPU load is %d%%" % cpuload() 

^{( Source code entirely )}

 cpuload .__ name __ == decorated
 cpuload () returns 16
 Executing cpuload took 105 ms
 CPU load is 16%

The @timed def cpuload(): ... unfolds in def cpuload(): ...; cpuload=timed(cpuload) def cpuload(): ...; cpuload=timed(cpuload) , so that as a result, the global name cpuload associated with the decorated function inside timed , closed to the original cpuload function through the variable fn . As a result, we see cpuload.__name__==decorated

As a decorator, you can use any expression whose value is a function that takes a function and returns a function. Thus, it is possible to create “decorators with parameters” (in fact, decorator factories):

 def repeat(times): """   times ,     """ def decorator(fn): def decorated2(*x): total = 0 for i in range(times): total += fn(*x) return total / times return decorated2 return decorator @repeat(5) def cpuload(): """   cpuload    """ print "cpuload.__name__==" + cpuload.__name__ print "CPU load is %d%%" % cpuload() 

^{( Source code entirely )}

 cpuload .__ name __ == decorated2
 cpuload () returns 7
 cpuload () returns 16
 cpuload () returns 0
 cpuload () returns 0
 cpuload () returns 33
 CPU load is 11%

The value of the expression repeat(5) is the decorator function, closed at times=5 . This value is used as a decorator; in fact, we have def cpuload(): ...; cpuload=repeat(5)(cpuload) def cpuload(): ...; cpuload=repeat(5)(cpuload)

You can combine several decorators on one function, then they are applied in a natural order - from right to left. If the two previous examples are combined into @timed @repeat(5) def cpuload(): - then we will get

 cpuload .__ name __ == decorated
 cpuload () returns 28
 cpuload () returns 16
 cpuload () returns 0
 cpuload () returns 0
 cpuload () returns 0
 Executing decorated2 took 503 ms
 CPU load is 9%

And if you change the order of decorators - @repeat(5) @timed def cpuload(): - then we get

 cpuload .__ name __ == decorated2
 cpuload () returns 16
 Executing cpuload took 100 ms
 cpuload () returns 14
 Executing cpuload took 109 ms
 cpuload () returns 0
 Executing cpuload took 101 ms
 cpuload () returns 0
 Executing cpuload took 100 ms
 cpuload () returns 0
 Executing cpuload took 99 ms
 CPU load is 6%

In the first case, the ad was expanded in cpuload=timed(repeat(5)(cpuload)) , in the second case - in cpuload=repeat(5)(timed(cpuload)) . Pay attention to the printed function names: you can trace the chain of calls in both cases.

The limiting case of parametric decoration is the decorator, which takes the decorator as a parameter :

 def toggle(decorator): """  ""  ""  """ def new_decorator(fn): decorated = decorator(fn) def new_decorated(*x): if decorator.enabled: return decorated(*x) else: return fn(*x) return new_decorated decorator.enabled = True return new_decorator @toggle(timed) def cpuload(): """   cpuload    """ print "cpuload.__name__==" + cpuload.__name__ print "CPU load is %d%%" % cpuload() timed.enabled = False print "CPU load is %d%%" % cpuload() 

^{( Source code entirely )}

 cpuload .__ name __ == new_decorated
 cpuload () returns 28
 Executing cpuload took 101 ms
 CPU load is 28%
 cpuload () returns 0
 CPU load is 0%

The value that controls the connection / disconnection of the decorator is stored in the enabled attribute of the decorated function: Python allows you to stick arbitrary attributes to any function.

The resulting toggle function can also be used as a decorator for decorators :

 @toggle def timed(fn): """   timed    """ @toggle def repeat(times): """   repeat    """ @timed @repeat(5) def cpuload(): """   cpuload    """ print "cpuload.__name__==" + cpuload.__name__ print "CPU load is %d%%" % cpuload() timed.enabled = False print "CPU load is %d%%" % cpuload() 

^{( Source code entirely )}

 cpuload .__ name __ == new_decorated
 cpuload () returns 28
 cpuload () returns 0
 cpuload () returns 0
 cpuload () returns 0
 cpuload () returns 0
 Executing decorated2 took 501 ms
 CPU load is 5%
 cpuload () returns 0
 cpuload () returns 16
 cpuload () returns 14
 cpuload () returns 16
 cpuload () returns 0
 Executing decorated2 took 500 ms
 CPU load is 9%

Um ... no, it didn't work! But why?
Why didn't the timed decorator shut down on the second cpuload call?

Recall that the global name timed is associated with a decorated decorator, i.e. with new_decorated function; it means that the timed.enabled = False line timed.enabled = False changes the attribute of the new_decorated function - the common “wrapper” of both decorators. It would be possible inside new_decorated instead of if decorator.enabled: to check if new_decorator.enabled: but then the line timed.enabled = False will disable both decorators at once.

Let's fix this bug: in order to use the enabled attribute on the “internal” decorator, as before, we new_decorated couple of methods on the new_decorated function:

 def toggle(decorator): """  ""  ""  """ def new_decorator(fn): decorated = decorator(fn) def new_decorated(*x): #   if decorator.enabled: return decorated(*x) else: return fn(*x) return new_decorated def enable(): decorator.enabled = True def disable(): decorator.enabled = False new_decorator.enable = enable new_decorator.disable = disable enable() return new_decorator print "cpuload.__name__==" + cpuload.__name__ print "CPU load is %d%%" % cpuload() timed.disable() print "CPU load is %d%%" % cpuload() 

^{( Source code entirely )}
The desired result was achieved - timed disconnected, but repeat continued to work:

 cpuload .__ name __ == new_decorated
 cpuload () returns 14
 cpuload () returns 16
 cpuload () returns 0
 cpuload () returns 0
 cpuload () returns 0
 Executing decorated2 took 503 ms 

 CPU load is 6%
 cpuload () returns 0
 cpuload () returns 0
 cpuload () returns 7
 cpuload () returns 0
 cpuload () returns 0
 CPU load is 1%

This is one of the most fascinating features of Python - not only attributes, but also arbitrary function methods can be added to functions. Functions on functions sit and functions chase.