We deploy nginx + mod_wsgi on the server
Hello For a long, long time I was looking at the wonderful django framework, reading a book , studying articles, trying to write hello worlds (it was easy and pleasant with the server built into the junga). And yesterday I tried to set up a combat server from the beginning to the end, and as it turned out, it was not so easy, and it even seemed to me that if I were younger and less experienced, I would have spat on this matter. So I decided to share with the readers a complete instruction, providing it with some arguments and configs. The article is designed for beginners, but it will be interesting to everyone, I promise.
There are several reasons for this. First, the solution must be fast, eat little memory and be sustainable. Nginx was created to be that way. In addition, according to performance tests , it is the nginx + wsgi bundle that gives less memory consumption and high fault tolerance under heavy loads. Secondly, it is known that the simpler the system, the more reliable it is. If we compare wsgi with fastcgi, we can say about the latter that this is a separate server on which the application itself is spinning. Someone should make sure that this server does not fall. wsgi is a call to python scripts from a C application (which is nginx), so by setting up the interface once, we reduce the number of entities twice and then communicate only with the web server.
In addition, I have an assumption that wsgi will be generally faster for the end user, although they are based only on theoretical reasoning: suppose that 50 users simultaneously connected to our server and requested equally heavy pages. In the case of fastcgi, nginx will immediately make 50 requests to the fastcgi server and will wait for a response. Purely theoretically identical requests that came at the same time will produce a bunch of running applications, it will be equal to competing for CPU time and all will be executed at once (assuming that the number of applications running by fastcgi is unlimited). Requests that come to wsgi, nginx will process itself, and since the maximum number of simultaneous requests that are executed is equal to the number of workers, they will queue up and users will receive the first answers almost immediately. But the latter will come in the same time as in the case of fastcgi, there's no way to go.
')
The most boring part, we will collect nginx. You need to collect it because there is simply no other way to connect mod_wsgi, so pull the latest stable version (now 0.7.60 ) from the Igor Sysoev website and unpack it somewhere.
The official mod_wsgi page, as I understand it, is here . And it is here that the main disappointment befalls us - the project has not been updated for more than a year. It is necessary to correct the source code for compatibility with the current version of nginx, but nothing, the main thing is that everything should start. Downloading the latest version of the module and also unpacking somewhere near the Indiana.
What specifically to edit:
1) In the archive itself with mod_wsgi, in the patches folder there is a patch nginx-0.6.x.patch , which patches the config and ngx_http_wsgi_module.c You can ignore the final message, we will not need it.
2) Since when was the last update of mod_wsgi, there was no current version of indiks, the official patch is not enough. Searches for other compilation errors led me only to this site . I don’t know Japanese, but it wasn’t necessary, the necessary patches are right on the page. True, the line numbers in this patch did not coincide with those in the source code, and I had to manually edit:
In the src / ngx_http_wsgi_module.c file, we declare somewhere closer to the beginning of the file the structure:
In the same place, the first call of the function ngx_garbage_collector_temp_handler is replaced with NULL.
Function call
change to
Looking ahead, I’ll say that later we will have to return to the source code mod_wsgi and fix something in them, but in principle they are already ready to build.
Navigate to the source directory of indjiniks and run
./configure --add-module = / path / to / mod_wsgi /
Here I had a little gag. The fact is that since the http_cache module began to be delivered with the indiniks, he began to demand openSSL. I searched for the openssl-dev package for a long time and even wanted to build it without this module, but then the package was found and it was just called ssl-dev. By the way, there is an error in the error description, a parameter that disables http_cache not “--without-http_cache”, but “--without-http-cache”.
Here I did make & make install.
The first thing to fix in nginx.conf is the number of workers. Probably you heard that usually the optimal number of workers is equal to the number of kernels on the machine. When using mod_wsgi, things change a little. The fact is that due to the fact that applications work in the context of these very workers, the number of simultaneously running processes is equal to the number of workers. Under ideal conditions, the number of workers in terms of the number of cores would load the processor at exactly 100%. But conditions are not perfect and application execution is sometimes interrupted. For example, when performing disk operations or requesting a database, if the database server is on another machine. So it seems to me that the optimal number of workers is twice the number of cores.
Further we configure lokheyshena in the server.
If we do as suggested in the example from mod_wsgi:
then we can immediately say goodbye to processing statics on the same server. Popular is the decision to set a regular file for popular file types and try to give them as files, and dump the rest on the backend. But I do not like this option. Firstly, when requesting a picture that is not on the server, there will be a standard message of an Ejinix, and secondly, what if you need to give an automatically generated pdf file? It is better that he also had a pdf extension. Therefore, I propose my own version: all the paths should first be searched on the disk, and if no such file is found, we follow the page to the backend (and if nothing is found on it either, there will be page 404 from the application, and not from the server).
It uses 2 files that will also need to be created:
From here we take wsgi_params :
And from the official django.py documentation:
It should be noted that in DJANGO_SETTINGS_MODULE, it is not the file name that is indicated, but the module name, i.e. for the given example, the real location of the configuration file should be “/path/to/project/settings.py”
If I missed nothing, then everything should be ready to launch the indiniks. We launch the browser, go to http: // localhost / (or whatever you have there) and see the welcome page of the junga, if the project is new, like mine, or the main page, if something is already written. Honestly, I already thought that the greatest difficulties were over, but it was not there. As soon as I typed http: // localhost / 1 (or any other url not from the root), I got the 500th indiniks. It is clear that this is worse than the five hundredth jung. The logs contained a record that the negative second argument is passed to the function PyString_FromStringAndSize (the length of the string).
After a showdown with the sources, I came to the conclusion that the reason was that the module was trying to remove the name of the locale from the beginning of the line path_info into the handler which got the url. Those. in our case, all the pages fall into the location backend and it is the 8 characters that the module tries to subtract from path_info, in which so much is not there. In the case when there were so many characters in the url, we got the path shortened from the beginning, i.e. instead of https: // localhost / 123456789 received https: // localhost89
The comment is worth TODO "we should check that clcf-> name is in r-> uri", i.e. “We have to check whether there is even the name of this location in the url line.
In principle, I understand why this was done, for example, it was possible to determine
and the url of the form http: // localhost / django / foo / bar / would be translated for jangi to https: // localhost / foo / bar /. But it is not clear why the author did not take into account that urls can be regexps and links in general (as in our case). I decided that I didn’t need such a buggy functionality, especially if I needed it, it can be done in django by rewriting urls.py c include.
In general, you need to fix in ngx_wsgi_run.c line 584:
after which, once again do make & make install, after extinguishing the working indiniks.
Python is not php. The application, once downloaded, is separated from its source code and does not know about its change. Therefore, we will have to restart the indiniiks from time to time when updating the code. Fortunately, indiniks is a very good server, we don’t have to write a script that would kill it and run it again, disconnecting all clients. It provides the ability to restart the workers gently, without instantly disconnecting the clients they served. This can be done by passing the HUP signal to the head process. Therefore, all we need is to write a simple script, calling it, say restart-server :
Each time you run this script, the application will be restarted. In addition, I would highly recommend hanging this script on crowns, say once an hour. On performance, this does not greatly affect, but the memory can save, if the scripts begin to flow.
Unfortunately, the resulting server has an Achilles heel. If suddenly it turns out that 10 people decide to load some kind of brake page at the same time (say, pdf generation or Avatar's compression), and you have only 10 workers, then no other request (even the easiest one, for example, gif picture) will be processed until not at least one worker is free. I unfortunately did not test other methods of communication, perhaps the same effect would be when working with fastcgi or http_pass, but it seems to me that since separate tcp connections are used there, they should not behave this way. But even this minus is not so terrible, because, as I said, the indiniks is a wonderful server. He divides the requests into heavy and light ones, and even if you have many heavy requests in the queue, then at least one worker will be released, he will spit out all the light requests first, and only then he will start heavy. In other words, light requests are always more important than heavy requests, in whatever order they come.
Why wsgi and nginx?
There are several reasons for this. First, the solution must be fast, eat little memory and be sustainable. Nginx was created to be that way. In addition, according to performance tests , it is the nginx + wsgi bundle that gives less memory consumption and high fault tolerance under heavy loads. Secondly, it is known that the simpler the system, the more reliable it is. If we compare wsgi with fastcgi, we can say about the latter that this is a separate server on which the application itself is spinning. Someone should make sure that this server does not fall. wsgi is a call to python scripts from a C application (which is nginx), so by setting up the interface once, we reduce the number of entities twice and then communicate only with the web server.
In addition, I have an assumption that wsgi will be generally faster for the end user, although they are based only on theoretical reasoning: suppose that 50 users simultaneously connected to our server and requested equally heavy pages. In the case of fastcgi, nginx will immediately make 50 requests to the fastcgi server and will wait for a response. Purely theoretically identical requests that came at the same time will produce a bunch of running applications, it will be equal to competing for CPU time and all will be executed at once (assuming that the number of applications running by fastcgi is unlimited). Requests that come to wsgi, nginx will process itself, and since the maximum number of simultaneous requests that are executed is equal to the number of workers, they will queue up and users will receive the first answers almost immediately. But the latter will come in the same time as in the case of fastcgi, there's no way to go.
')
From theory to practice
The most boring part, we will collect nginx. You need to collect it because there is simply no other way to connect mod_wsgi, so pull the latest stable version (now 0.7.60 ) from the Igor Sysoev website and unpack it somewhere.
The official mod_wsgi page, as I understand it, is here . And it is here that the main disappointment befalls us - the project has not been updated for more than a year. It is necessary to correct the source code for compatibility with the current version of nginx, but nothing, the main thing is that everything should start. Downloading the latest version of the module and also unpacking somewhere near the Indiana.
What specifically to edit:
1) In the archive itself with mod_wsgi, in the patches folder there is a patch nginx-0.6.x.patch , which patches the config and ngx_http_wsgi_module.c You can ignore the final message, we will not need it.
2) Since when was the last update of mod_wsgi, there was no current version of indiks, the official patch is not enough. Searches for other compilation errors led me only to this site . I don’t know Japanese, but it wasn’t necessary, the necessary patches are right on the page. True, the line numbers in this patch did not coincide with those in the source code, and I had to manually edit:
In the src / ngx_http_wsgi_module.c file, we declare somewhere closer to the beginning of the file the structure:
static ngx_path_init_t ngx_http_wsgi_temp_path = {
ngx_string(NGX_HTTP_WSGI_TEMP_PATH), { 1, 2, 0 }
};In the same place, the first call of the function ngx_garbage_collector_temp_handler is replaced with NULL.
Function call
ngx_conf_merge_path_value(conf->temp_path,
prev->temp_path,
NGX_HTTP_WSGI_TEMP_PATH, 1, 2, 0,
ngx_garbage_collector_temp_handler, cf);change to
ngx_conf_merge_path_value(cf, &conf->temp_path,
prev->temp_path,
&ngx_http_wsgi_temp_path);Looking ahead, I’ll say that later we will have to return to the source code mod_wsgi and fix something in them, but in principle they are already ready to build.
Navigate to the source directory of indjiniks and run
./configure --add-module = / path / to / mod_wsgi /
Here I had a little gag. The fact is that since the http_cache module began to be delivered with the indiniks, he began to demand openSSL. I searched for the openssl-dev package for a long time and even wanted to build it without this module, but then the package was found and it was just called ssl-dev. By the way, there is an error in the error description, a parameter that disables http_cache not “--without-http_cache”, but “--without-http-cache”.
Here I did make & make install.
Nginx configuration
The first thing to fix in nginx.conf is the number of workers. Probably you heard that usually the optimal number of workers is equal to the number of kernels on the machine. When using mod_wsgi, things change a little. The fact is that due to the fact that applications work in the context of these very workers, the number of simultaneously running processes is equal to the number of workers. Under ideal conditions, the number of workers in terms of the number of cores would load the processor at exactly 100%. But conditions are not perfect and application execution is sometimes interrupted. For example, when performing disk operations or requesting a database, if the database server is on another machine. So it seems to me that the optimal number of workers is twice the number of cores.
Further we configure lokheyshena in the server.
If we do as suggested in the example from mod_wsgi:
location / {
wsgi_pass /path/to/nginx/django.py;
}then we can immediately say goodbye to processing statics on the same server. Popular is the decision to set a regular file for popular file types and try to give them as files, and dump the rest on the backend. But I do not like this option. Firstly, when requesting a picture that is not on the server, there will be a standard message of an Ejinix, and secondly, what if you need to give an automatically generated pdf file? It is better that he also had a pdf extension. Therefore, I propose my own version: all the paths should first be searched on the disk, and if no such file is found, we follow the page to the backend (and if nothing is found on it either, there will be page 404 from the application, and not from the server).
location / {
root /path/to/www/;
error_page 404 = @backend;
log_not_found off;
}
location = / {
#
wsgi_pass /path/to/nginx/django.py;
include wsgi_params;
}
location @backend {
wsgi_pass /path/to/nginx/django.py;
include wsgi_params;
}It uses 2 files that will also need to be created:
From here we take wsgi_params :
wsgi_var REQUEST_METHOD $request_method;
wsgi_var QUERY_STRING $query_string;
wsgi_var CONTENT_TYPE $content_type;
wsgi_var CONTENT_LENGTH $content_length;
wsgi_var SERVER_NAME $server_name;
wsgi_var SERVER_PORT $server_port;
wsgi_var SERVER_PROTOCOL $server_protocol;
wsgi_var REQUEST_URI $request_uri;
wsgi_var DOCUMENT_URI $document_uri;
wsgi_var DOCUMENT_ROOT $document_root;
wsgi_var SERVER_SOFTWARE $nginx_version;
wsgi_var REMOTE_ADDR $remote_addr;
wsgi_var REMOTE_PORT $remote_port;
wsgi_var SERVER_ADDR $server_addr;
wsgi_var REMOTE_USER $remote_user;And from the official django.py documentation:
import os, sys
# ,
sys.path.append('/path/to/')
#
os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()It should be noted that in DJANGO_SETTINGS_MODULE, it is not the file name that is indicated, but the module name, i.e. for the given example, the real location of the configuration file should be “/path/to/project/settings.py”
Big bummer
If I missed nothing, then everything should be ready to launch the indiniks. We launch the browser, go to http: // localhost / (or whatever you have there) and see the welcome page of the junga, if the project is new, like mine, or the main page, if something is already written. Honestly, I already thought that the greatest difficulties were over, but it was not there. As soon as I typed http: // localhost / 1 (or any other url not from the root), I got the 500th indiniks. It is clear that this is worse than the five hundredth jung. The logs contained a record that the negative second argument is passed to the function PyString_FromStringAndSize (the length of the string).
After a showdown with the sources, I came to the conclusion that the reason was that the module was trying to remove the name of the locale from the beginning of the line path_info into the handler which got the url. Those. in our case, all the pages fall into the location backend and it is the 8 characters that the module tries to subtract from path_info, in which so much is not there. In the case when there were so many characters in the url, we got the path shortened from the beginning, i.e. instead of https: // localhost / 123456789 received https: // localhost89
The comment is worth TODO "we should check that clcf-> name is in r-> uri", i.e. “We have to check whether there is even the name of this location in the url line.
In principle, I understand why this was done, for example, it was possible to determine
location /django {
wsgi_pass …
}and the url of the form http: // localhost / django / foo / bar / would be translated for jangi to https: // localhost / foo / bar /. But it is not clear why the author did not take into account that urls can be regexps and links in general (as in our case). I decided that I didn’t need such a buggy functionality, especially if I needed it, it can be done in django by rewriting urls.py c include.
In general, you need to fix in ngx_wsgi_run.c line 584:
if (clcf->name.len == 1 && clcf->name.data[0] == '/') {if (1 || (clcf->name.len == 1 && clcf->name.data[0] == '/')) {after which, once again do make & make install, after extinguishing the working indiniks.
Server restart
Python is not php. The application, once downloaded, is separated from its source code and does not know about its change. Therefore, we will have to restart the indiniiks from time to time when updating the code. Fortunately, indiniks is a very good server, we don’t have to write a script that would kill it and run it again, disconnecting all clients. It provides the ability to restart the workers gently, without instantly disconnecting the clients they served. This can be done by passing the HUP signal to the head process. Therefore, all we need is to write a simple script, calling it, say restart-server :
sudo kill -HUP `cat /usr/local/nginx/logs/nginx.pid`Each time you run this script, the application will be restarted. In addition, I would highly recommend hanging this script on crowns, say once an hour. On performance, this does not greatly affect, but the memory can save, if the scripts begin to flow.
A spoon of tar
Unfortunately, the resulting server has an Achilles heel. If suddenly it turns out that 10 people decide to load some kind of brake page at the same time (say, pdf generation or Avatar's compression), and you have only 10 workers, then no other request (even the easiest one, for example, gif picture) will be processed until not at least one worker is free. I unfortunately did not test other methods of communication, perhaps the same effect would be when working with fastcgi or http_pass, but it seems to me that since separate tcp connections are used there, they should not behave this way. But even this minus is not so terrible, because, as I said, the indiniks is a wonderful server. He divides the requests into heavy and light ones, and even if you have many heavy requests in the queue, then at least one worker will be released, he will spit out all the light requests first, and only then he will start heavy. In other words, light requests are always more important than heavy requests, in whatever order they come.
Source: https://habr.com/ru/post/62545/
All Articles