Python Web Development through the eyes of a PHP programmer

Introduction

The article would like to raise the issues of differences in the use of Python for web-development compared with PHP. I hope the article will not lead to holivars, since it is not at all about which language is better or worse, but only about the technical features of Python.

A little about the languages ​​themselves

PHP is a web-based language, created to die (in a good sense of the word). From a low-level point of view, a PHP application is rather a set of separate scripts possible with a single semantic entry point.

Python is a universal programming language that also applies to the web. From a technical point of view, a Python web application is a full-fledged application loaded into memory, having its own internal state, saved from a request to a request.
')
Based on the above features, there are differences in error handling in web applications. In PHP, there is a whole zoo of error types (errors, exceptions), not all of which can be intercepted, although this (impossibility of interception) does not matter much, since the application lives exactly as long as one request is processed. An uncaught error simply results in an early exit from the handler, and the removal of the application from memory. A new request will be processed by a new “clean” application. In Python, the application is always in memory, processing many requests without "reloading". Therefore, maintaining the correct, predictable state of the application is crucial. All errors use the standard exception mechanism and can be intercepted (with the possible exception of SyntaxError). An uncaught error will terminate the application, which will need to be restarted from the outside.

There are many ways to "prepare" PHP and Python for the web. Next, I will focus on the two most familiar to me (and it seems the most popular) - PHP + FastCGI ( php-fpm ) and Python + WSGI ( uWSGI ). Of course, in front of both these bundles a front end server is supposed to exist (for example, Nginx).

Python multithreading support

Starting an application server (for example, uWSGI) causes the Python interpreter to be loaded into memory, and then the web application itself. Usually, the bootstrap module of an application imports the modules it needs, makes initialization calls and eventually exports a prepared callable object that conforms to the WSGI specification. As you know, when you first import Python modules, the code inside them is executed, including creating and initializing variables. Between two consecutive HTTP requests, the state of the interpreter is not reset, therefore the values ​​of all the variables of the module level are saved.

Let's write the simplest WSGI application that demonstrates the above with an example:

n = 0 def app(env, start_response): global n n += 1 response = "%.6d" % n start_response('200 OK', [('Content-Type', 'text/plain')]) return [bytes(response, 'utf-8')] 

Here n is a module variable and it will be created with a value of 0 when the application is loaded into memory with the following command:

 uwsgi --socket 127.0.0.1:8080 --protocol http --single-interpreter --processes 1 -w app:app 

The application itself simply displays the value of the variable n . For avid PHP programmers, it looks meaningless, since “it should” display the “000001” line every time .

Let's do a test:

 ab -n 500 -c 50 http://127.0.0.1:8080/ curl http://127.0.0.1:8080 

As a result, we get the string "000501" , which confirms our statement that the application is loaded into the memory of uwsgi and maintains its state between requests.

If you run uWSGI with the --processes 2 parameter and run the same test, then several consecutive curl calls will show that we already have 2 different incremental sequences. Since ab sends 500 requests, about half of them fall on one uWSGI process, and the rest on the second. The expected values ​​returned by curl will be approximately "000220" and "000280" . The Python interpreter, apparently, is one per process, and we have 2 independent environments and real parallel query processing (in the case of a multi-core processor).

Python supports threads as part of the language. The classic implementation (CPython) uses native OS threads, but there is a GIL — only one stream is executed at a time. However, race condition problems are still possible, since even n + = 1 is not an atomic operation.


Increasing the number of threads without waiting for IO in the code of HTTP request handlers does not accelerate processing (or rather, it slows it down, since threads can “push” by switching contexts). The real parallelism of the threads as a result of the constraints of GIL is not created, although they are not green threads, but real OS threads.

Let's do another test. Let's run uwsgi with 1 process, but 10 handler threads in it:

 uwsgi --socket 127.0.0.1:8080 --protocol http --single-interpreter --processes 1 --threads 10 -w app:app 

and execute ab 5000 queries to the application.

 ab -n 5000 -c 50 http://127.0.0.1:8080/ 

Subsequent queries of curl 127.0.0.1 : 8080 will show that we have only one incremental sequence, the value of which is <= 5000 (less than 5000 it may be in the case of race condition on an increment).

Language Impact on Application Architecture

Each HTTP request is processed in a separate stream (true for processes, since the process has at least 1 stream). At the same time, each thread during its lifetime (which, under ideal conditions, coincides with the lifetime of the entire uwsgi application) processes a large number of HTTP requests, preserving its state (ie, the values ​​of the module level variables) from request to request. This is almost the main difference from the HTTP request processing model in PHP, where each request comes in a new, newly initialized environment and the application load must be performed again each time.

A typical approach in large web applications in PHP is to use the Dependency Injection Container to control initialization and access to the application services level. A good example is Pimple . For each HTTP request, the first step is the initialization code, which registers all available services in the container. Then, as needed, access is made to the services object (lazy) in the controllers. Each service may depend on other services, dependencies are resolved again through the container in the initialization code of the service aggregate.

 //   $container['session_storage'] = function ($c) { return new SessionStorage('SESSION_ID'); }; $container['session'] = function ($c) { return new Session($c['session_storage']); }; //   class MyController { public function __construct() { // get the session object $this->session = $container['session']; // "" ,    } } 

Thanks to the container, you can ensure the one-time creation of objects and the return of ready-made objects for each subsequent access to the service (if necessary). But this magic only works within a single HTTP request, so services can easily be initialized with request-specific values. Such values ​​are often the current authorized user, the session of the current user, the HTTP request itself, etc. At the end of the request, the services will still be destroyed, and at the beginning of processing the next request, new ones will be created and initialized. In addition, you can hardly worry about memory leaks if processing one HTTP request fits into the limits assigned to the script, since the creation of services takes place on demand (lazy) and for one request each required service will most likely be created only in a single copy.

Now, taking into account the flow model of Python described above, you can see that using a similar approach in the Python web application is not possible without additional efforts. If the container is a variable of the module level (which looks quite logical), all the services it contains cannot be initialized with values ​​specific to the current request, since the service will be a shared resource between several streams that process several HTTP pseudo-requests. parallel. At first glance, there are two ways to deal with this problem — to make service objects independent of the current HTTP request (the service method calls remain dependent, and the stack variables used in the methods are not shared resources) or make the container a resource of the flow rather than process (then each stream will only communicate with its independent set of services, and at one point in time one stream can process only one HTTP request).

The seeming plus of the first approach is that services are initialized only once in the lifetime of the uwsgi process. It is also possible to save memory (since we have only one set of services for all threads). On the other hand, the processing of a specific HTTP request requires only some (most likely small) sub-capabilities of all available services. If the application is large enough and has an impressive number of services, then after processing a certain number of HTTP requests, the vast majority of services will be initialized and stored in the process memory. It looks like this may be a serious problem.

The second approach can be implemented using threading.local . So, for example, comes flask. To illustrate the approach, you can implement a stream-local storage for some events:

 class EventsStore: def __init__(self): self._store = threading.local() def add(self, event): self.get_all().append(event) def clear(self): if self.has(): del self._store.events def get_all(self): if not self.has(): self._store.events = [] return self._store.events def has(self): return hasattr(self._store, 'events') def pop_event(self): return self._store.events.pop() if self.has() and self._store.events else None 

Similarly, scoped_session in SQLAlchemy is implemented, which allows you to have a unique connection to the base for each thread.

In the case of the second approach, memory problems can be avoided by destroying the container at the end of each request and creating a new container before processing the new request. This is very similar to the PHP query processing model. Minus - the container and services are initialized for each new request (the same code is constantly executed), i.e. not taking advantage of Python threads.

Conclusion

Languages ​​are different - approaches are different. Python has its own characteristics and it’s not completely clear to me how to use them correctly. Python provides the ability to have the “state” of an application and multi-threaded request processing, but you have to pay a potentially possible race condition (in the light of the fact that even n + = 1 is not an atomic operation) and possibly a more complex application architecture. I would like to hear how bearded Python-developers build their web-applications.