Synchronous code in asynchronous Twisted, or a tale about how to cross a hedgehog with a snake

All is well

Twisted is an asynchronous (event-oriented) framework written in Python. A powerful tool for the rapid development of network (and not only) services. It is designed using the Reactor design pattern. Services created using Twisted are fast and reliable, the framework allows you not to write macaroni code rich with incomprehensible callbacks, it has beautiful helpers inside (Deferred, Transport, Protocol etc). In other words, the backend of developers makes our life better.

But there are problems

The main problem is that numerous, reliable, tested, convenient libraries that use basically Python synchronous modules (socket, os, ssl, time, select, thread, subprocess, sys, signal etc) will simply take and block our main process. , the reactor cycle and trouble will come. Such libraries, for example, are psycopg2, request, mysql and others. In particular, psycopg2 is used in Django ORM as one of the backend databases.

What to do?

There are three ways. Difficult, acceptable and good. Difficult - to implement an analog library on Twisted. Acceptable - use deferToThread and run synchronous code in separate threads (using the thread pool implemented in Twisted). About a good way (in my opinion) and will be discussed in the article.





')

Use green streams and events to switch context!

What do we need for this?

• Greenlets are lightweight “green” threads that run inside the main application process.
• Gevent - a framework that allows you to switch the context between the greenlets, at the moment when the executable code is blocked
• Reactor Method [deferToGreenlet], which allows you to wrap a greenlet in Deferred

An example of the use of technology in a real project

I did not write my own implementation of the reactor with the ability to send code to the Greenleta, as I found a ready-made solution, tested and implemented it in the project. The reactor code can be taken from here .



To use geventreactor when initializing an application, you need to install it:

from geventreactor import install install() 

Now new methods are available to us:

 __all__ = ['deferToGreenletPool', 'deferToGreenlet', 'callMultipleInGreenlet', 'waitForGreenlet', 'waitForDeferred', 'blockingCallFromGreenlet', 'IReactorGreenlets', 'GeventResolver', 'GeventReactor', 'install'] 

Similar to reactor.deferToThread (f, * args, ** kwargs), you can call reactor.deferToGreenlet (f, * args, ** kwargs), where f is a callable object, and * args and ** kwargs are its arguments.



To make it work, you must also patch the libraries in the namespace:

 from gevent import monkey monkey.patch_all() 

After these manipulations, the main Python libraries will be patched by the Gevent framework. See the Gevent documentation.



Now all libraries or code that directly imports them, when calling blocking methods or functions, will trigger the corresponding events in the Gevent system. Callbacks are hung for these events, allowing you to switch the context between the greenlets.



My project uses Django ORM to manipulate data in PostgreSQL. Therefore, in order for the ORM methods not to block the process, you need to use a special backend that allows you to create a pool of database connections and switch between connections. One of the backends is django-db-geventpool



Using django-db-geventpool is not difficult. It is enough to follow the documentation.

What's next?

Method reactor.deferToGreenlet returns a Deferred object with which you can work as with ordinary Deferred.



For example, we have a model:

 class ExampleModel(models.Model): title = models.CharField(max_length=256) 

We want to get all the models and pass them on to some handler inside the system. We can write something like:

 d = reactor.deferToGreenlet(ExampleModel.objects.all) 

And our code will not block the main process. Indeed, at the moment when Django ORM calls cursor.execute (), which will wait for a response from the database driver, geventreactor switches the context to another Deferred.

What is the result?

We can execute synchronous code inside Twisted without creating unnecessary threads or processes without blocking the event loop of the reactor. The main thing is to follow the basic principles of working with asynchronous systems, pieces of code should not be run for too long, gevent allows you to forcefully switch the context from anywhere in the code, where it is convenient for us, just call gevent.sleep ().