Django in non-blocking style, or in pursuit of the Holy Grail

A tip

When reading about Twisted, Tornado, Node.js, many python programmers have a question - “but if you take and rewrite Django in a non-blocking style?”. The usual answer to this question is no, you will not wait. And the truth is, to rewrite a whole macaroni-kolbechny framework, you need a lot of strength and a lot of enthusiasm. Writing with callbacks, a very dubious pleasure.

It would have been, but as I wrote in my last note, there are greenlet-s in the python-world, which with a flick of the wrist helps to hide all these asynchronous moments from the programmer, to a certain extent. At the time of writing this note, I was skeptical about the idea of ​​making Django work in non-blocking mode - well, really, who will take this?


')

The essence

But in fact, everything is not so scary, watch your hands - we create a WSGI resource in Twisted, start listening to HTTP requests; we give each request to the Django WSGI handler, wrapping it in a greenlet; any requests that can be converted to non-blocking are passed to Twisted, which will kick our greenlet at the moment data is ready; is ready!



And what do we have such that you can turn into a non-blocking mode in the first place? In Django, the main such site is the database. Is it possible to communicate with it asynchronously? Absolutely right, but you need a special driver.

For a long time in Twisted there is such - PGAsync. He has serious problems, he has long been abandoned, but who prevents us from picking up the fallen banner? I made such an attempt, at the moment the driver is more stable than the original, but the work is not yet finished. However, we can get an asynchronous database for Django, and it will be Postgresql (oh, sorry, Mysql users, but who's stopping you from writing your own?). The driver is used in the current project being developed by Imarto Networks imarto.net , and it was decided to post it for free use. I laid it out before readiness, only to demonstrate the asynchronous Django, be lenient.

To use it in Django, we need to write our database backend. I fiddled with something from the back end for psycopg2, wrapping calls to the database driver in the function of transferring control to the main greenlet, that is, the main program flow. The standard wsgi-resource from twisted.web also had to be slightly redone, because now it should throw processing requests not into the pool of threads, but into greenlet.

How to feel

Sources can be obtained here, there is a test django-project - bitbucket.org/deepwalker/tx_green
Sources of database drivers - bitbucket.org/deepwalker/tx-postgresql

To use it, you must place pgasync, green_wsgi.py, tx_green.py, tx_postgresql in a place where the interpreter can find them. Then in the test_green_dj directory, configure the settings for connecting to the database, ./manage.py syncdb, twistd -n - y server.py. Everything, at 127.0.0.1 : 8001, we meet our test project.

About tests

The tests that I conducted related mainly to the verification of work, I did not measure the performance gain. If someone spends his time on this good deed, I will be grateful if you share the results.