The uWSGI Spooler

When designing scalable systems, where you have to access a variety of external components, for example, using a third-party API, sending mail or converting video, the best way to implement is an asynchronous model with a queue system that is the link to the interaction of all system components.

The most popular queuing system in Python is Celery, it has a wide range of task management capabilities. Unfortunately, Celery-based systems are difficult to maintain in a healthy state, and when something goes wrong, it is not easy to find a problem. You can ask any devops about your experience with Celery, but be prepared to hear not very pleasant words.

Fortunately, there is an alternative solution - uWSGI Spooler, and in this article I will tell about it in more detail.

The main difference from Celery is that it does not need to use additional components (Celery itself and storage, for example Redis), so the number of points of failure is reduced by two. As a repository of tasks can be used a directory, external directory or network pool.

To control Python programs, we often use uWSGI. Why? Because it is easy to configure, reliable, flexible and meets most requirements.

In addition to serving Python code in the form of providing continuous access to a web application, uWSGI includes a Spooler component that implements a queuing system. Spooler has some features, and the documentation on it is quite scarce.

Using uWSGI Spooler is easy, just one-two-three! But there are a few nuances.
The uwsgi module cannot be imported from the code, and the code cannot be tested from the console accordingly, you need to run the uwsgi worker every time, for which you need to create a config:

[uwsgi] socket = /var/run/mysite.sock master = True processes = 4 project_dir = /home/myuser/mysite chdir = %(project_dir) spooler = /var/uwsgi_spools/mysite_spool spooler-import = path.to.spool.package # (package to import spool file) spooler-frequency = 10 # Frequency for scanning spool max-requests = 5000 module = wsgi:application touch-reload = wsgi.py 

Worker file:

 from uwsgidecorators import spool, uwsgi @spool def my_func(args): print(args) # do some job 

Setting a problem from your code:

 import uwsgi_spools.mysite_spool as mysite_spool mysite_spool.my_func.spool(test=True) 

As you can see from the example, the threshold for entering is very low.

Inside the task there is a single argument that contains a dictionary with three service keys (function name ud_spool_func , task name spooler_task_name, task status ud_spool_ret ) and all the parameters that were passed when creating the task, for example, the test key.

Task can return three statuses:

• -2 (SPOOL_OK) - task completed, will be removed from the queue;
• -1 (SPOOL_RETRY) - something went wrong, the task will be recalled;
• 0 (SPOOL_IGNORE) - ignore task.

All other values ​​will be interpreted as -1 (SPOOL_RETRY).

Feature: the @spool decorator @spool executed once (returns SPOOL_OK) if the function did not fall with the exception.
In order to manage the life cycle you need to use @spoolraw .

Special keys (auxiliary) when creating a task:

• spooler - the absolute path to the spooler that will perform the task;
• at - unix time, when the task should be executed (more correctly, it will not be executed earlier than this value);
• priority - indicates a subfolder in the task folder (a greater number of workers can be allocated to such a subfolder), using --spooler-ordered, you can set priorities;
• body - this key is used for values ​​greater than 64 KB, the task will be available in serialized form.

In addition to the @spool decorator, the @spool decorator is @timer , which takes the number of seconds as an argument and allows you to perform the function to be decorated at a specified interval.

 @timer(30) def my_func(args): print(args) # do some job every 30 sec 

Similar to @timer there is the @timer decorator, which will restart the execution of the function (completion of the task with the status SPOOL_RETRY).

 @spoolforever def my_func(args): print(args) # do some job and repeat 

To set up workers to work over the network, you need to add the address where it will be available in the ini-file:

 socket = 127.0.0.1:10001 

When creating a task, specify the address of the recipient of the task:

 uwsgi.send_message(“127.0.0.1:10001”, 17, 0, test=True, 5) #  uwsgi.spool(test=True, spooler=“127.0.0.1:10001”) 

Thus, uWSGI Spooler can be used as a substitute for queues, but if you still lack opportunities or want some sugar, you can use uwsgi-tasks , which implements the missing.