Deferred: all the details
The previous article described the basic principles of Deferred and its use in asynchronous programming. Today we will try to examine in detail the operation of Deferred and examples of its use.
So, Deferred is a delayed result, a result of execution that will become known after a while. The result stored in Deferred may be an arbitrary value (successful execution) or an error (exception) that occurred during the execution of an asynchronous operation. Since we are interested in the result of the operation and we received from some asynchronous function Deferred, we want to perform actions at the moment when the result of the execution will be known. Therefore, in addition to the result, Deferred also stores a chain of handlers: result handlers (callback) and error handlers (errback).
Consider in more detail the chain of handlers:
')
The handlers are arranged in “layers” or levels, the execution takes place clearly in the levels from top to bottom. At the same time, callback and errback handlers are located at each level, one of the elements may be missing. At each level, either callback or errback can be executed, but not both. Execution of handlers occurs only once, there can be no re-entry.
The callback handler functions are functions with one argument — the result of the execution:
The errback handler functions accept an exception wrapped into the Failure class as a parameter:
Deferred execution begins with the result that appears in Deferred: successful execution or exception. Depending on the result, the corresponding handler branch is selected: callback or errback. After this, the nearest level of handlers is searched, in which there is a corresponding handler. In our example in the figure, a successful execution result was obtained and the result was passed to the
Further execution leads to calling handlers at lower levels. If callback or errback is completed by returning a value that is not Failure, the execution is considered successful and the result is sent to the input of the callback handler at the next level. If, during the execution of the handler, an exception was thrown or a value of the Failure type was returned, errback will be passed to the next level, which will receive the exception as a parameter.
In our example, the
Like all other Python objects, a Deferred object lives as long as it is referenced from other objects. Usually the object that returned Deferred saves it, because he needs to transfer the result to Deferred after the asynchronous operation is completed. Most often, other participants (adding event handlers) do not save references to Deferred, so the Deferred object will be destroyed at the end of the chain of handlers. If Deferred is destroyed, in which an unhandled exception remains (execution ended with an exception and there are no more handlers), a debug message is printed on the screen with a traceback exception. This situation is similar to “bouncing off” an unhandled exception to the top level in a regular synchronous program.
The return value of callback and errback can also be another Deferred, then the execution of the chain of current Deferred handlers is suspended until the end of the chain of handlers of the nested Deferred.
In the example shown in the figure, the callback2 handler returns not the usual result, but another Deferred - Deferred2. At the same time, the execution of the current Deferred is suspended until the result of the execution of Deferred2 is obtained. The result of Deferred2 - successful or exception - becomes the result passed to the next level of handlers of the first Deferred. In our example, Deferred2 ended with an exception that will be input to the
Each errback exception handler is an analogue of a try..except block, and the except block usually responds to some type of exception, this behavior is very simple to reproduce using Failure :
The
Note that the errback handler must complete in one of two ways:
There is a third option - to return Deferred, then the further execution of handlers will depend on the result of Deferred.
In our example, we processed the exception and passed 0 as the result (for example, the absence of some key is equivalent to its zero value).
As soon as asynchrony appears, that is, some function returns Deferred instead of an immediate value, asynchrony begins to propagate through the function tree above, forcing Deferred to return from functions that were previously synchronous (returning the result directly). Consider a conditional example of such a transformation:
If for some reason the function
But now the function
A similar “transformation” scheme occurs with real functions: we get the results in the form of Deferred from the functions that are located in the tree, we hang our callback handlers on their Deferred, which correspond to the old, synchronous code of our function, if we had exception handlers, we add and errback handlers.
In practice, it is better to first identify those code points that will be asynchronous and will use Deferred, than to convert synchronous code into asynchronous. Asynchronous code starts with those calls that cannot build the result directly:
In the process of writing an application, it is often clear that at this point there will be asynchronous access, but it is not there yet (the interface with the DBMS, for example) is not implemented. In this situation, you can use the functions
If we do not know whether the called function will return the result synchronously or return Deferred, and do not want to depend on its behavior, we can wrap the call to it in
Such a variant of the function
You can talk about Deferred for a very long time, as an additional reading, I can again recommend the list of materials at the end of the previous article .
So, Deferred is a delayed result, a result of execution that will become known after a while. The result stored in Deferred may be an arbitrary value (successful execution) or an error (exception) that occurred during the execution of an asynchronous operation. Since we are interested in the result of the operation and we received from some asynchronous function Deferred, we want to perform actions at the moment when the result of the execution will be known. Therefore, in addition to the result, Deferred also stores a chain of handlers: result handlers (callback) and error handlers (errback).
Consider in more detail the chain of handlers:
')
The handlers are arranged in “layers” or levels, the execution takes place clearly in the levels from top to bottom. At the same time, callback and errback handlers are located at each level, one of the elements may be missing. At each level, either callback or errback can be executed, but not both. Execution of handlers occurs only once, there can be no re-entry.
The callback handler functions are functions with one argument — the result of the execution:
def callback(result): … The errback handler functions accept an exception wrapped into the Failure class as a parameter:
def errback(failure): … Deferred execution begins with the result that appears in Deferred: successful execution or exception. Depending on the result, the corresponding handler branch is selected: callback or errback. After this, the nearest level of handlers is searched, in which there is a corresponding handler. In our example in the figure, a successful execution result was obtained and the result was passed to the
callback1 handler.Further execution leads to calling handlers at lower levels. If callback or errback is completed by returning a value that is not Failure, the execution is considered successful and the result is sent to the input of the callback handler at the next level. If, during the execution of the handler, an exception was thrown or a value of the Failure type was returned, errback will be passed to the next level, which will receive the exception as a parameter.
In our example, the
callback1 handler was successful, its result was passed to the callback2 handler, in which an exception was thrown, which led to a chain of errback handlers, at the third level, the errback handler was missing and the exception was passed to errback4 , which handled the exception, returned successful the execution result, which is now the result of Deferred, but no more handlers. If another level of handlers is added to Deferred, they will be able to access this result.Like all other Python objects, a Deferred object lives as long as it is referenced from other objects. Usually the object that returned Deferred saves it, because he needs to transfer the result to Deferred after the asynchronous operation is completed. Most often, other participants (adding event handlers) do not save references to Deferred, so the Deferred object will be destroyed at the end of the chain of handlers. If Deferred is destroyed, in which an unhandled exception remains (execution ended with an exception and there are no more handlers), a debug message is printed on the screen with a traceback exception. This situation is similar to “bouncing off” an unhandled exception to the top level in a regular synchronous program.
Deferred squared
The return value of callback and errback can also be another Deferred, then the execution of the chain of current Deferred handlers is suspended until the end of the chain of handlers of the nested Deferred.
In the example shown in the figure, the callback2 handler returns not the usual result, but another Deferred - Deferred2. At the same time, the execution of the current Deferred is suspended until the result of the execution of Deferred2 is obtained. The result of Deferred2 - successful or exception - becomes the result passed to the next level of handlers of the first Deferred. In our example, Deferred2 ended with an exception that will be input to the
errback2 handler of the first Deferred.Errback Exception Handling
Each errback exception handler is an analogue of a try..except block, and the except block usually responds to some type of exception, this behavior is very simple to reproduce using Failure :
def errback(failure): """ @param failure: (), failure @type failure: C{Failure} """ failure.trap(KeyError) print "Got key error: %r" % failure.value return 0 The
trap method of the Failure class checks whether the exception wrapped in it is a heir or directly the KeyError class. If this is not the case, the original exception is again thrown out, interrupting the execution of the current errback, which results in the transfer of control to the next errback in the chain of handlers, which simulates the behavior of the except block when the type of exception does not match (control is passed to the next block). The value property holds the original exception, which you can use to get additional information about the error.Note that the errback handler must complete in one of two ways:
- Return some value that will be the input value of the next callback, which means that the exception was handled.
- Throw out the original or new exception - the exception was not processed or a new exception was re-thrown, the chain of errback handlers continues.
There is a third option - to return Deferred, then the further execution of handlers will depend on the result of Deferred.
In our example, we processed the exception and passed 0 as the result (for example, the absence of some key is equivalent to its zero value).
Preparing for asynchrony in advance
As soon as asynchrony appears, that is, some function returns Deferred instead of an immediate value, asynchrony begins to propagate through the function tree above, forcing Deferred to return from functions that were previously synchronous (returning the result directly). Consider a conditional example of such a transformation:
def f(): return 33 def g(): return f()*2 If for some reason the function
f cannot return the result immediately, it will start returning Deferred: def f(): return Deferred().callback(33) But now the function
g to return Deferred, hooking to the chain of handlers: def g(): return f().addCallback(lambda result: result*2) A similar “transformation” scheme occurs with real functions: we get the results in the form of Deferred from the functions that are located in the tree, we hang our callback handlers on their Deferred, which correspond to the old, synchronous code of our function, if we had exception handlers, we add and errback handlers.
In practice, it is better to first identify those code points that will be asynchronous and will use Deferred, than to convert synchronous code into asynchronous. Asynchronous code starts with those calls that cannot build the result directly:
- network I / O;
- access to network services database, memcached;
- Remote RPC calls
- operations that will be allocated to the thread in the Worker model, etc.
In the process of writing an application, it is often clear that at this point there will be asynchronous access, but it is not there yet (the interface with the DBMS, for example) is not implemented. In this situation, you can use the functions
defer.success or defer.fail to create Deferred, which already contains the result. Here's how to rewrite function f short: from twisted.internet import defer def f(): return defer.success(33) If we do not know whether the called function will return the result synchronously or return Deferred, and do not want to depend on its behavior, we can wrap the call to it in
defer.maybeDeferred , which any variant would make equivalent to calling Deferred: from twisted.internet import defer def g(): return defer.maybeDeferred(f).addCallback(lambda result: result*2) Such a variant of the function
g will work with both synchronous and asynchronous f .Instead of conclusion
You can talk about Deferred for a very long time, as an additional reading, I can again recommend the list of materials at the end of the previous article .
Source: https://habr.com/ru/post/52924/
All Articles