Battle server for Django application: Ubuntu Server 10.04 LTS + django 1.4 + nginx + gunicorn

Many Django development tutorials reveal how to quickly get a working debug server (python manage.py runserver), and the issue of deploying in combat mode often remains undisclosed or covers far from the simplest and most effective methods.
Below I will talk about one of the ways to deploy a site on Django in combat mode, starting with the choice of hosting, ending with the deployment of a web server. Thus, the article may be useful to those who have mastered the development based on Django, but have no experience deploying servers. My way is one of many, but it is quite simple, efficient in work and easy to maintain. We use VPS hosting, Ubuntu 10.04, nginx, gunicorn.

Table of contents

• Table of contents
• Project structure
• Hosting
• Ubuntu server
• Deploying the project
• Nginx web server setup
• Gunicorn and supervisor setup
• Troubleshooting
• Conclusion
• useful links

')

Project structure

To further the story, it is necessary to tell about the structure of the project being developed:

- myproject/ -  ,  ,      ; - env/ -  ,    virtualenv; - src/ -   ,      IDE,  ,     myproject  ; - myproject/ -   ,    Django-; - __init__.py - settings.py - manage.py - urls.py - myapp1/ - myapp2/ ... - static_content/ -   ,   -   (  - nginx),  ,    - ,   ; - static/ -    (    ); - media/ - media-,        ; - docs/ -    , ,  ; - logs/ - ,   ,   ; - pids/ -  pid-. 

At first glance, the nesting depth of the source may seem redundant. However, my experience shows that the costs of working at such a depth are small, and the merits of greater structuredness are considerable.

Hosting

For myself, I found out a long time ago that the most convenient hosting for django is VPS hosting. Having tried several VPS hosting sites, I settled on the most acceptable price / performance ratio - FirstVDS . I use the tariff for 400 rubles per month - 1500 MB RAM, 24000 MB HDD. But at first it cost 150 rubles (in this case you will need to teach some applications to consume less resources).
So, in order to start you need to order and pay for VPS, while choosing the type of virtualization OpenVZ - an incomplete imitation of a physical machine, but quite sufficient for our tasks, it is worth noting that with such virtualization for the same money they give more memory useful for memcached . As an operating system, I recommend choosing Ubuntu Server 10.04 LTS #. Ubuntu Server 12.04 LTS is also available, I managed to put it into operation without much difficulty, however, the hosting template has appeared recently, and the hosting provider’s experts do not recommend it yet be installed on the combat servers. Now I use it only on a test server.



After ordering and paying for the mail comes SSH-access, you can now begin to configure the operating system.

Ubuntu server

Having access, I, first of all, update the system, then disable root access via SSH.
You can connect via SSH in Windows with the well-known Putty program, but I use its more “wired” modification - kitty . As the IP address of the server, enter the address sent by the host, connect, enter root as user and password when prompted.
To connect to Linux-like systems, we use a native terminal (Terminal, Konsole, etc.), by typing a simple command in it:

 ssh root@1.2.3.4 

where 1.2.3.4 is the IP address sent by the hoster, when prompted, enter the password sent by the hosting provider.
First of all, we will change the password of the root, for the sake of paranoia:

 passwd 

upon request, we enter the new password twice, I advise you to generate and store in KeePassX , do not hesitate to make it more complicated (> 20 characters), you will hardly have to use it.
We update packages in the system:

 apt-get update apt-get upgrade 

Add a user on behalf of whom we will deploy the server:

 adduser myuser 

you can not enter anything except the password, but it is better to come up with it more complicated
Now you need an editor to edit configs, I prefer nano:

 apt-get install nano 

Add the rights to the user to execute commands as root (sudo):

 nano /etc/sudoers #   root ALL=(ALL:ALL) ALL #     myuser ALL=(ALL:ALL) ALL 

Now we disable the ability for the root user to log in via SSH (optionally, a security measure):

 nano /etc/ssh/sshd_config #  PermitRootLogin yes #   PermitRootLogin no 

after that, I reboot the machine to make sure everything went as it should:

 reboot 

After a minute, you can log in as myuser (or you can pre-verify that the username root does not work anymore):

 ssh myuser@1.2.3.4 

Now I put a number of system packages necessary for the Django application and infrastructure to work:

 sudo apt-get install gcc mysql-server python-mysqldb memcached mercurial python-profiler w3m python-setuptools libmysqlclient-dev git-core python-dev rabbitmq-server supervisor nginx 

Let's take a look at what is what:

• gcc is a C compiler, C ++ compiler, something else before the heap, definitely needed to build some packages;
• python-setuptools - installing the easy_install command, you only need to install virtualenv, the fact is that the system package virtualenv is not installed the latest version, with which there are a number of annoying problems;
• mysql-server - from the name it is clear that this is a MySQL server, we will need it, since we are going to raise the database server on the host of the web application;
• libmysqlclient-dev - this package is needed in order to build the MySQL access interface for python;
• python-dev - python's API, definitely needed to build some packages;
• supervisor is a daemon controller program, needed for taming a web server and possibly a number of other programs;
• nginx - nginx web server, I express my gratitude to Igor Sysoev ;
• rabbitmq-server - optional, AMQP server, useful if you need to use celery;
• memcached is an optional, in-memory known cache implementation;
• git-core - optional, the client for the git version control system, will be required if the necessary packages are stored in the git-repository;
• mercurial - optionally, the client for the version control system mercurial (hg) is required if the necessary packages are stored in the mercurial repository.

Deploying the project

For deployment, we will use the virtual environment using the virtualenv tool.
In the previous chapter, we installed the useful utility easy_install, with which we will install the latest version of virtualenv now:

 sudo easy_install virtualenv 

Note. I deliberately did not install pip globally, so that there were no random conflicts with it in pip's in virtual environments.
Now you can begin to deploy the project. First you need to place it somewhere in your home directory, let it be:

 /home/myuser/web/myproject 

I use the git version control system. But you can achieve this by simply copying over the scp. In Windows, you can use WinSCP , in Linux file managers support the address format sftp: //myuser@1.2.3.4, as you probably guessed, data for access via SSH is used as authentication data.
In my project, the root file for building a virtual environment is build_env.sh:

 #!/bin/bash echo $0: Creating virtual environment virtualenv --prompt="<myenv>" ./env mkdir ./logs mkdir ./pids mkdir ./db mkdir ./static_content mkdir ./static_content/media echo $0: Installing dependencies source ./env/bin/activate export PIP_REQUIRE_VIRTUALENV=true ./env/bin/pip install --requirement=./requirements.conf --log=./logs/build_pip_packages.log echo $0: Making virtual environment relocatable virtualenv --relocatable ./env echo $0: Creating virtual environment finished. 

and the requirements.conf file, which contains the packages necessary for the application to work:

 django git+git://github.com/sehmaschine/django-grappelli.git#egg=django-grappelli git+git://github.com/django-mptt/django-mptt.git#egg=django-mptt git+git://github.com/krvss/django-social-auth.git#django-social-auth git+git://github.com/gabrielhurley/django-wymeditor.git#django-wymeditor git+git://github.com/jtauber/django-mailer.git#django-mailer git+git://github.com/tweepy/tweepy.git#tweepy django-celery django-debug-toolbar django-pdb python-memcached MySQL-python xlrd unidecode anyjson gunicorn pillow south fabric requests xlwt 

This is a possible set of packages, somehow I will tell you more about some of them, but we do not delve into the development aspects of this topic.
So, after running in the project folder / home / myuser / web / myproject:

 ./build_env.sh 

set a virtual environment for our server.
Now you need to take care of creating the database. In our projects we use MySQL. We start the MySQL console:

 mysql -uroot -pROOTPASSWORD 

where we create the base and user:

 CREATE DATABASE myproject CHARACTER SET utf8 COLLATE utf8_general_ci; CREATE USER 'myproject'@'localhost' IDENTIFIED BY 'USERPASSWORD'; GRANT ALL PRIVILEGES ON myproject.* TO 'myproject'@'localhost'; 

Now access settings can be entered into the project settings file (settings.py or local_settings.py).
If you have a database dump, you know what to do. If not, then create in it the scheme and the necessary data:

 cd ~/web/myproject #     source env/bin/activate #    python manage.py syncdb #      

now we will collect static files:

 python manage.py collectstatic 

On this application is ready to run.

Nginx web server setup

To configure the Nginx web server, you need to place the myproject.conf config (the suffix must be .conf) in the / etc / nginx / sites-available / folder.
You can create and edit a file in the SSH console over nano:

 sudo nano /etc/nginx/sites-available/myproject.conf 

then paste the contents of the example below (Shift + Insert), fix them to fit your needs, save (Ctrl + O), exit (Ctrl + X).
An example of the content of the myproject.conf file:

 upstream myproject.ru { server localhost:12345 fail_timeout=0; } server { listen 80; server_name www.myproject.ru; rewrite ^/(.*) http://myproject.ru/$1 permanent; } server { listen 80; client_max_body_size 4G; server_name myproject.ru; access_log /home/myuser/web/myproject/logs/myproject.access.log; keepalive_timeout 5; root /home/myuser/web/myproject/static_content; location / { proxy_pass http://myproject.ru; } error_page 500 502 503 504 /500.html; location = /500.html { root /home/myuser/web/myproject/static_content/static/html; } location ~ ^/(static|media)/ { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect off; if (!-f $request_filename) { proxy_pass http://myproject.ru; break; } } } 

Let's try to understand the general words, what is what.
In this example, the upstream is a localhost: 12345 socket, to which we will forward all requests that do not satisfy the following url:
/ static / * - static site files;
/ media / * - media files, files that are created during the operation of the site;
On this socket we will hang on listening to our gunicorn in the next part of the article. The name of the Upstream, theoretically, is chosen arbitrarily, however, I encountered problems if the name of the Upstream was different from the domain name of the site (I do not remember the text of Django errors to describe in more detail).

Another interesting part of the config:
Redirecting from myproject.ru subdomain www.myproject.ru ; you can do the opposite, if you prefer;
5XX error handling is provided by issuing a static 500.html file. It is quite simple to get such a file: make the site show a 404 page, save it as 500.html in the browser and edit the error text;
if the static file is not found, the request is sent to gunicorn so that it can produce a standard error 404.

After the config file is placed where it should be (you can check with cat /etc/nginx/sites-available/myproject.conf), you need to put a symlink on it:

 sudo ln -s /etc/nginx/sites-available/myproject.conf /etc/nginx/sites-enabled/ 

and restart nginx:

 sudo service nginx restart 

Gunicorn and supervisor setup

When deploying the project, we installed the gunicorn package, now we use it to generate dynamic HTTP responses (HTTP response) by a Django application. Gunicorn itself can receive requests and formulate responses to them, however, it is necessary to keep gunicorn running, for example, to start when the system is restarted, we will entrust it to the supervisor specializing in this program.
Note that gunicorn developers strongly do not recommend putting it on the front line, but instruct the reception of requests from users to specialized servers, such as nginx, as we do.
So, we create in the /etc/supervisor/conf.d/ folder the file myproject.conf with the following content:

 [program:myproject] command=/home/myuser/web/myproject/env/bin/python /home/myuser/web/myproject/src/myproject/manage.py run_gunicorn --bind=localhost:12345 --workers=3 --pid=/home/myuser/web/myproject/pids/gunicorn.pid --log-file /home/myuser/web/myproject/logs/gunicorn.log directory=/home/myuser/web/myproject/src/myproject umask=022 autostart=true autorestart=true startsecs=10 startretries=3 exitcodes=0,2 stopsignal=TERM stopwaitsecs=10 user=myuser 

Note that in the gunicorn parameters we pass the same port for wiretapping (in our example 12345), which was indicated in the nginx config. The number of workflows is set to 3, selected this parameter subjectively for one of the sites, based on their workloads, and so far I cannot recommend any value.
After the config is placed in the correct folder (you can check it by running cat /etc/supervisor/conf.d/myproject.conf) you need to re-read the file or just restart supervisor, I use the second option for reliability:

 sudo supervisorctl reload 

You can check the launch success with the command:

 sudo supervisorctl status 

if successful, the status of the myproject process will become RUNNING after a few seconds.

Troubleshooting

Conclusion

Note the positive aspects of this deployment:

• Deploying the solution on a VPS virtual machine gives you virtually unlimited administration.
• The independence of nginx from gunicorn, as is often suggested when using Apache and mod_python, for example. In particular, it is quite easy to replace, for example for debugging, with a gunicorn development server:
  

   sudo supervisorctl stop myproject python manage.py runserver localhost:12345 #     

  
  Now nginx will send requests to the developer's server, without even knowing about the changes.
• Nginx as a distributor of static files is recognized as one of the best, maybe even the best.
• virtualenv allows you to easily and naturally isolate the virtual environments of different projects from each other. Nowadays, disk space costs are incomparably cheaper than messing around with package version conflicts.
• gunicorn under the control of the supervisor is much more convenient and reliable than the ways of launching gunicorn under the control of scripts in /etc/init.d/ and some other ways of launching, since it provides a way of monitoring running programs, carefully restarting programs in case of failures.

In the article only one of the many options for deploying a Django application was examined, I came to it through many trials and errors. My recommendations do not pretend to be the ultimate truth. Moreover, I will be glad to any constructive criticism.

useful links

1. FirstVDS - cheap VPS hosting, with a link order for a quarter cheaper.
2. Putty and Kitty are SSH clients for Windows.
3. WinSCP - SCP-client under Windows, no need to raise the FTP-server, just use your existing SSH-account.
4. KeePassX - cross-platform password storage utility.
5. Ubuntu LTS is a release release policy for Ubuntu Long Term Support.
6. Ubuntu Security - setting for paranoids for the desktop, but something can be learned for the server.
7. Django Deployement - other ways to deploy sites on Django.
8. virtualenv - use virtualenv.
9. Nginx - a lot of interesting things about configuring nginx.
10. Supervisor + gunicorn - about using gunicorn via nginx.