Do it yourself web service with asynchronous queues and parallel execution

Everyone should do their job efficiently and on time. Suppose you need to make a web service for classifying pictures based on a trained neural network using the caffe library. Nowadays, quality is asynchronous non-blocking calls, the possibility of parallel execution of several tasks in the presence of free processor cores, monitoring task queues ... The RQ library allows you to implement all this in a short time without studying a ton of documentation.

Let's make a web service on one server, focused on lightly loaded projects and relatively long tasks. Naturally, its application is not limited to these your neural networks.

Formulation of the problem

Input data is a file (for example, a picture in JPEG format). For simplicity, we assume that it has already been placed in a dedicated directory. The output is a string in JSON format. For solidity, we will use the standard HTTP result codes.

The web service will implement two HTTP calls (let's call this API):

• / process / [file name] - process the file (preload the input file into the selected directory, return the task ID)
• / result / [task identifier] - get the result (if the result is not ready, return the code 202 "Not ready", if the result is ready - return json, if the identifier of the task does not exist, return the code 404 "Not found")

Installing components in Ubuntu

We will use Flask as the HTTP server. Installation:

If pip is not installed:

sudo apt-get install python-pip sudo apt-get install --upgrade pip 

Actually, the installation of Flask:

 pip install flask 

Now you need to install Redis - the data warehouse and message broker:

 wget http://download.redis.io/redis-stable.tar.gz tar xvzf redis-stable.tar.gz cd redis-stable make sudo make install 

Installing the RQ (Redis Queue) library:

 pip install rq 

For automatic start and configuration of all components we will use Supervisor :

 sudo apt-get install supervisor 

We write our service

It is easy in Flask. Create a deep_service.py file:

 #  ,      BASEDIR = '/home/sergey/verysecure' #   import argparse import os import json #       from flask import Flask app = Flask(__name__) from redis import Redis from rq import Queue # ,    ,     #classify.py - ,      caffe #,       from classify import main #    Redis q = Queue(connection=Redis(), default_timeout=3600) #    API @app.route('/process/<path:file_path>') def process(file_path): full_path = os.path.join(BASEDIR, file_path) #     BASEDIR argv = {'input_file': full_path, 'gpu': True} args = argparse.Namespace(**argv) r = q.enqueue_call(main, args=(args,), result_ttl=86400) return r.id #   :  4-     , #   JSON   numpy def decimal_default(obj): if isinstance(obj, float32): return round(float(obj), 4) else: raise TypeError() #    API @app.route('/result/<id>') def result(id): try: job = q.fetch_job(id) if job.is_finished: return json.dumps(job.result, ensure_ascii=False, default=decimal_default) else: return 'Not ready', 202 except: return "Not found", 404 if __name__ == '__main__': app.run() #app.run(debug=False, host='0.0.0.0') 

Manual start - check how it works

At this stage, you can check whether our web service is running. Run Redis:

 redis-server 

We start one workflow (they can be run several, for example, by the number of processor cores or by the presence of several video cards if they are required for data processing). It is better to start the process from the directory in which computing functions will be launched, in our case, this is where classif.py lies:

 rq worker 

We start the http-server:

 python deep_service.py 

Write the cat.jpg image to the input data directory and execute the service request:

 wget 127.0.0.1/process/cat.jpg 

In response, we get the task ID. Copy the identifier and execute the second request to the service:

 wget 127.0.0.1/result/[] 

In response, we get a JSON string with weights of the image belonging to IMAGENET categories.
Now it remains to configure the automatic launch of our server components.

Autostart

Setting up a supervisor is probably the hardest part of this journey. A good tutorial on setting up a supervisor is here .

First of all, you need to understand that supervisor starts every process in its own environment. In most cases of complex calculations, the program that implements them depends on a number of settings, for example, paths. These settings are usually stored in the file /home/usersname/.bashrc

For example, the caffe library of neural network computing and the Python modules to it required adding the following lines to this file:

 export PATH=/usr/local/cuda-7.5/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig export PKG_CONFIG_PATH LD_LIBRARY_PATH=/home/username/caffe/build/lib:$LD_LIBRARY_PATH export PYTHONPATH="${PYTHONPATH}:/home/username/caffe/python" 

Copy these lines to the clipboard!

In the / usr / local / bin directory create a deep_worker.sh file

 #!/bin/bash cd /home/username/caffe/python export PATH=/usr/local/cuda-7.5/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig export PKG_CONFIG_PATH LD_LIBRARY_PATH=/home/username/caffe/build/lib:$LD_LIBRARY_PATH export PYTHONPATH="${PYTHONPATH}:/home/username/caffe/python" rq worker 

Well, you understand - in the first line we move to the working directory, then we insert the environment variables copied from .bashrc, then we start the process.

In the / usr / local / bin directory create a deep_flask.sh file

 #!/bin/bash cd /home/username/caffe/python export PATH=/usr/local/cuda-7.5/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig export PKG_CONFIG_PATH LD_LIBRARY_PATH=/home/username/caffe/build/lib:$LD_LIBRARY_PATH export PYTHONPATH="${PYTHONPATH}:/home/username/caffe/python" python deep_service.py 

Again - in the first line we move to the working directory, then paste the environment variables copied from .bashrc, then start our Flask-server.

Some system administration:

 sudo chmod +x /usr/local/bin/deep_flask.sh sudo chmod +x /usr/local/bin/deep_worker.sh mkdir /var/log/deepservice 

In the /etc/supervisor/conf.d directory, create a deepservice.conf file:

 [program:redis] command=/usr/local/bin/redis-server autostart=true autorestart=true stderr_logfile=/var/log/deepservice/redis.err.log stdout_logfile=/var/log/deepservice/redis.out.log [program:worker1] command=/usr/local/bin/deep_worker.sh autostart=true autorestart=true stderr_logfile=/var/log/deepservice/worker1.err.log stdout_logfile=/var/log/deepservice/worker1.out.log user=username directory=/home/username/caffe/python [program:flask] command=/usr/local/bin/deep_flask.sh autostart=true autorestart=true stderr_logfile=/var/log/deepservice/flask.err.log stdout_logfile=/var/log/deepservice/flask.out.log user=username directory=/home/username/caffe/python 

Finally, run this whole construct:

 sudo supervisorctl reread sudo supervisorctl update 

Everything!