The shortest recording of asynchronous calls in tornado or patch bytecode in the decorator

A complex asynchronous handler in tornado sometimes sprawls into dozens of callback functions, which makes it difficult to perceive and modify code. Therefore, there is a tornado.gen module that allows you to write a handler as a generator. But a lot of yield gen.Task (...) also does not look very. Therefore, in a fit of delirium, I wrote a simplifying decorator:

Before	After
@asynchronous @gen.engine def get(self): result, status = yield gen.Task( db.users.find_one, { '_id': ObjectId(user_id), }, )	@asynchronous @gen.engine @shortgen def get(self): result, status << db.users.find_one_e({ '_id': ObjectId(user_id), }, )

How it works

As you already noticed, we replaced yield with << . Since python will not allow us to do this by standard means, we need to modify the bytecode. For simple work with it we will use the Byteplay module. Let's see the bytecode of two simple functions:

from byteplay import Code from pprint import pprint def gen(): a = yield 1 pprint(Code.from_code(gen.func_code).code)	[(SetLineno, 5), # 5 (LOAD_CONST, 1), # 1 (YIELD_VALUE, None), # "" (STORE_FAST, 'a'), # a (LOAD_CONST, None), (RETURN_VALUE, None)]
def shift(): a << 1 pprint(Code.from_code(shift.func_code).code)	[(SetLineno, 10), (LOAD_GLOBAL, 'a'), # a (LOAD_CONST, 1), # 1 (BINARY_LSHIFT, None), # a (POP_TOP, None), # (LOAD_CONST, None), (RETURN_VALUE, None)]

Therefore, we will make a simple patcher especially for this situation:

from byteplay import YIELD_VALUE, STORE_FAST code = Code.from_code(shift.func_code) code.code[3] = (YIELD_VALUE, None) code.code[4] = (STORE_FAST, 'a') code.code.pop(1) pprint(code.code)

[(SetLineno, 10), (LOAD_CONST, 1), (YIELD_VALUE, None), (STORE_FAST, 'a'), (LOAD_CONST, None), (RETURN_VALUE, None)]

Now we have a bytecode almost identical to the bytecode of the gen function, apply it to shift and check the result:

shift.func_code = code.to_code() res_gen = gen().send(None) res_shift = shift().send(None) print res_gen print res_shift print res_gen == res_shift

1 1 True

The result was the same. The code for the general situation can be viewed on github . You can learn more about baytkod in official documentation . In the meantime, we return to the tornado. Take the finished decorator shortgen . And we will write a simple handler:

 def fetch(callback): callback(1) class Handler(BaseHandler): @asynchronous @gen.engine @shortgen def get(self): result << gen.Task(fetch) 

The code is a little better, but we still have to manually wrap the call in gen.Task , so we will create another decorator to automate this process:

 def fastgen(fnc): return partial(gen.Task, fnc) @fastgen def fetch(callback): callback(1) class Handler(BaseHandler): @asynchronous @gen.engine @shortgen def get(self): result << fetch() 

Now everything looks quite decent, but how will it work with third-party libraries? No way, so now we need to patch them! No, we are not going to patch bytecode now, but apply just the monkey patch. In order not to break the old code, we will replace __getattribute__ from the necessary classes with:
')

 def getattribute(self, name): attr = None if name.find('_e') == len(name) - 2: attr = getattr(self, name[:-2]) if hasattr(attr, '__call__'): return fastgen(attr) else: return super(self.__class__, self).__getattribute__(name) 

Now, if the patched object does not have an attribute, for example, find_e (the _e postfix is ​​added so as not to break the old code), the find attribute wrapped in the fasttgen decorator will be returned to us.
And now the code, for example, for asyncmongo, will look like this:

 from asyncmongo.cursor import Cursor Cursor.__getattribute__ = getattribute class Handler(BaseHandler): @asynchronous @gen.engine @shortgen def get(self): result, status << self.db.posts.find_e({'name': 'post'}) 

How to use it

First, install the resulting module:

 pip install -e git+https://github.com/nvbn/evilshortgen.git#egg=evilshortgen 

Now we patch the classes we need:

 from evilshortgen import shortpatch shortpatch(Cls1, Cls2, Cls3) 

Let’s decorate our own asynchronous methods and functions:

 from evilshortgen import fastgen @fastgen def fetch(id, callback): return callback(id) 

And use in the handler:

 from evilshortgen import shortgen class Handler(BaseHandler): @asynchronous @gen.engine @shortgen def get(self, id): data << fetch(12) num, user << Cls1.fetch() 

Known Issues

The call can only set values ​​for variables:

 a << fetch() #  self.a << fetch() #   

Complex unpacking is not supported:

 a, b << fetch() #  (a, b), c << fetch() #   

Links

Evilshortgen on github
Details about baytkod
Byteplay