Asynchronous Python 3.5 and Mongodb

This is a rather loose translation of an article about the main innovations of the asynchronous mongodb driver used in tornado . The main motive for writing this translation is the innovations introduced in this version, such as support for asyncio , async , await and Python 3.5 . The article itself is not how many listing of innovations, how many concise examples of asynchronous work with MongoDB .


Introduction
asyncio
aggregate
Python 3.5
async and await

Introduction

Recently a new Beta version of Python driver for Mongodb , Motor has been published. This version contains one of the biggest updates. For installation you can use:
python -m pip install --pre motor==0.5b0

Motor 0.5 still depends on PyMongo 2.8.0 . This is an outdated version of PyMongo , but now there was not enough time to fully upgrade to the third version, which is excusable, since this release is quite large.
')

asyncio

Motor can now integrate with asyncio , as an alternative to Tornado . Big thanks to Remi Joleen, Andrei Svetlov svetlov and Nikolai Novik for their great contribution to Motor integration for working with asyncio .

APIs Tornado and asyncio are related. Motor Example with Tornado :

 # Tornado API from tornado import gen, ioloop from motor.motor_tornado import MotorClient @gen.coroutine def f(): result = yield client.db.collection.insert({'_id': 1}) print(result) client = MotorClient() ioloop.IOLoop.current().run_sync(f) 

And here is an example for asyncio:

 import asyncio from motor.motor_asyncio import AsyncIOMotorClient @asyncio.coroutine def f(): result = yield from client.db.collection.insert({'_id': 1}) print(result) client = AsyncIOMotorClient() asyncio.get_event_loop().run_until_complete(f()) 

Unlike Tornado , asyncio does not include the http implementation, much less is a framework. To do this, use the aiohttp library of Andrey Svetlov. A small example for working Motor with aiohttp .

aggregate

MotorCollection.aggregate now returns the cursor by default, and the cursor is returned directly without yield . The old syntax is no longer supported:

 # Motor 0.4 and older, no longer supported. cursor = yield collection.aggregate(pipeline, cursor={}) while (yield cursor.fetch_next): doc = cursor.next_object() print(doc) 

In Motor 0.5 just do:

 # Motor 0.5: no "cursor={}", no "yield". cursor = collection.aggregate(pipeline) while (yield cursor.fetch_next): doc = cursor.next_object() print(doc) 

In asyncio this uses yield from :

 # Motor 0.5 with asyncio. cursor = collection.aggregate(pipeline) while (yield from cursor.fetch_next): doc = cursor.next_object() print(doc) 

Python 3.5

Now Motor compatible with Python 3.5 , which required some effort. It was difficult because Motor not only works with coroutines, it uses coroutines inside it to implement some of its functions, such as MotorClient.open and MotorGridFS.put .
There was a method for writing coroutines that work in Python 2.6 with Python 3.4 , but in Python 3.5 this was completely broken. There is no uniform way to return values ​​from Python 3.5 native coroutine or Python 2 generators based on a coroutine, so all motor internal coroutines that return values ​​were rewritten using callbacks.

async and await

The reward for the effort spent on integrating with Python 3.5 is that motor now works with a native coroutine written with the async keywords and the await syntax:

 async def f(): await collection.insert({'_id': 1}) 

The cursor from MotorCollection.find , MotorCollection.aggregate , or MotorGridFS.find can be beautifully and very effectively integrated in native coroutines with async for :

 async def f(): async for doc in collection.find(): print(doc) 

How effective? For a collection of 10,000 documents, this sample code was executed in 0.14 seconds.

 # Motor 0.5 with Tornado. @gen.coroutine def f(): cursor = collection.find() while (yield cursor.fetch_next): doc = cursor.next_object() print(doc) 

The following code, in which gen.coroutine and yield simply replaced by async and await , performs about the same.

 # Motor 0.5 with Tornado, using async and await. async def f(): cursor = collection.find() while (await cursor.fetch_next): doc = cursor.next_object() print(doc) 

But with async for , the runtime takes 0.04 seconds, that is, three times faster.

 # Motor 0.5 with Tornado, using async for. async def f(): cursor = collection.find() async for doc in cursor: print(doc) 

However, MotorCursor in to_list still plays the main role:

 # Motor 0.5 with Tornado, using to_list. async def f(): cursor = collection.find() docs = await cursor.to_list(length=100) while docs: for doc in docs: print(doc) docs = await cursor.to_list(length=100) 

The function with to_list is twice as fast as asynchronous, but it doesn’t look so beautiful and requires specifying the size of the chunk. I think that async for looks quite stylish and works fast enough to use it in most cases.

Beta versions of motor releases were published by no means always, but this time in a different way. The integration of asyncio into motor is completely new. And since it required widespread refactoring of the motor core, and rewriting the existing tornado integration, a beta version was released in order to correct all the omissions.

PS Request for grammatical errors and translation errors to write to the PM.