We automate the layout of django-projects on the server
Setting up VDSs for displaying django projects is quite tiring, and it's easy to forget something (because you don't do it every day). Much better when this process is automated: with less effort, you can get a properly configured project and a set of commands to work with it.
There are different approaches to this process: python-specific ( fabric , buildout ) or non-specific ( puppet , Chef , sets of shell scripts, etc.).
The fabric approach is a locally executed script that walks ssh to the server and executes commands there. This approach is quite straightforward and easy to debug, and so good ( review on Habré). From the various teams of the fabric, a bike called django-fab-deploy gradually emerged. This is a set of fabric scripts that can configure servers under Debian Lenny or Squeeze, and then with minimal efforts deploy django projects there and manage these projects in the future.
')
With the release of Debian, Squeeze took up django-fab-deploy more seriously, corrected some rough edges and now, I think, it's about time to tell about this project. The project has documentation , there will be a brief retelling with lyrical digressions.
General principles:
Server part:
The main criteria for choosing a deployment method were stability / reliability, and here the bundle apache + mod_wsgi + nginx is currently not equal. I think it’s not difficult to add support for ubunt, but what can I say about this, since there is no support for ubunt right now (I don’t use ubunta myself).
A project would be well kept in a version control system (mercurial is now supported ). There is a half-plug with loading / unpacking tar.gz in case the project is not in VCS; it works, but in the production mode I did not check it (there was no need for it), and there is a chance that some features will appear.
The following describes the basic way of deploying the server, if for some reason it does not fit, then it is not scary: django-fab-deploy is just a set of scripts; They can be used in parts, or any utility to peep, or send a patch, or just read the article.
It is assumed that we have a clean Debian server (it is possible and not “clean”, more on that later), ssh access via the public key to root is configured.
Please do not use VPS on OpenVZ, because they are limited to VIRT instead of RSS and therefore a lot of software (including Apache and mysql + innodb) under OpenVZ eats much more memory than under xen / kvm (10 times - yes, easily).
Django projects often use some kind of trick with 'local_settings.py', lie in the version control system, have a file with pip dependencies in the repository and a folder with settings for the web server. Django-fab-deploy doesn’t require anything other than that. Further details.
1. Install django-fab-deploy and its dependencies:
django-fab-deploy does not work with Fabric 0.9.x, Fabric must be installed from github.
2. Create a file fabfile.py in the project root. This file should contain one or more functions that configure server parameters. Well, because this is a regular fabric script, there may be any other (project-specific) commands. Example:
Pay attention to env.hosts. There is a feature: specify as the user (my_site) the name of your project (it must be the correct name of the variable in python - without dashes and spaces). It doesn't matter if there is no such user, django-fab-deploy will be able to create it and configure ssh access later. In any case, having a separate user for each project is a good idea.
In env.conf, some conventions apply to simplify everything. For example, the database name (DB_NAME) by default is the same as the name of the project instance (INSTANCE_NAME), which, in turn, is the same as the user from env.hosts.
To learn more about what and how you can configure here, see the documentation for fabfile api .
3. Create customization templates for the web server, etc .:
The config_templates folder will appear in the project root, there are settings templates. Projects are different, and often the settings will require changes - at least fluently read them.
In the settings templates, you can use variables enclosed in {{}} (the template engine here is jinja2). These variables will be replaced with values ​​from the env.conf dictionary when laying out templates on the server.
4. Create a config.server.py in the project root. This file will become the config.py file on the server. Example:
Create also config.py with local settings for development. Import config.py to settings.py:
The trick is widely known, often config.py also called local_settings.py. This file should be added to ignored in VCS (if VCS is used).
As you can see, config.server.py is also a template (like the files from the config_templates folder), variables from env.conf will also be inserted into it.
5. Create the 'reqs' folder in the project root. An example of such a folder can be obtained by executing the following command from the project root:
This folder should contain text files with pip dependencies. One file has a special meaning: reqs / all.txt. This is the main file with dependencies, all packages from there will be installed during deployment. There you can either list all the dependencies, or (better) add other files with dependencies via the “-r” option.
There is also the
After steps 1-5 have been completed, the project should look something like this:
If so, then the project is ready.
1. If a non-existent user was specified in env.hosts, then create it and configure ssh access by key, manually or like this (you will need a file with your public key):
Check that ssh works:
2. Set up the database. If the database is already configured manually (for example, the project is already running), then nothing needs to be done. django-fab-deploy now can install mysql and create an empty database for a project:
mysql_install does nothing if mysql is already installed on the server. If mysql is not installed, then it is set, and the password for the mysql root user is set to env.conf ['DB_PASSWORD'].
mysql_create_db creates an empty database called env.conf ['DB_NAME'] (in our example, this will be the user name from env.hosts).
If you are using not mysql or the user mysql is not root, then it is better to do everything by hand now. Better yet, automate (write a fabric script for this) and send the patch to django-fab-deploy.
3. Everything, the server is ready. You can cross your fingers and run a command from the project root:
If everything was done correctly, the site should earn.
This command:
Source codes will be in ~ / src / <INSTANCE_NAME>, virtualenv will be placed in ~ / envs / <INSTANCE_NAME>.
django-fab-deploy will disable the “default” sites for Apache and nginx, and will also command Apache ports.conf (Apache will no longer listen to port 80). If other sites were running on the server under Apache, they will become unavailable because of this. If only nginx stuck out, then everything should be fine - django-fab-deploy doesn't do anything tricky with the server.
Upload changes to the server and apply them:
Another example (put the changes on the prod server, while updating the dependencies and performing the migrations):
Update web server settings:
Update janga settings (config.server.py):
A complete list of commands can be found in the documentation. If you want something more high-level (in the spirit - launched fab redeploy and everything was immediately updated, and the code, settings, and dependencies, and migrations were completed) - you can easily write your fab-command as a wrapper over basic commands. If the push command does too much (it, for example, runs tests by default), then do not hesitate - look at the code and write a more suitable version in your fabfile.py. If you consider any such experiment useful and successful - open the ticket in the bug tracker .
The closest analogue to django-fab-deploy is woven . Apparently, also a great thing. Woven is focused on Ubuntu, Debian is supported roughly like Ubuntu in django-fab-deploy: "it seems like it should work with minor changes, but no one knows for sure." We started to do everything at about the same time, at first there was some kind of game in the classes, then they simplified everything. They have advanced far away. On the other hand, in django-fab-deploy is several times smaller than the source code, it is smaller and simpler, and will remain so.
Many people do similar projects. I searched recently for a githab / bitback, and I added 11 more analogs to djangopackages (different degrees of elaboration), you can look here at a comparative tablet: djangopackages.com/grids/g/deployment
Current documentation can always be found here: packages.python.org/django-fab-deploy
Source code and bug tracker repository : bitbucket.org/kmike/django-fab-deploy
Connect: use, correct, write comments, suggestions, how to improve everything, etc.
The description in the article refers to version 0.4 and may be outdated; read, if possible, documentation.
It is not necessary to use django-fab-deploy, you can write your own scripts with 0, use woven, buildout or whatever. But nevertheless, before writing everything with 0, it is better, of course, to do “homework” and see how existing projects are implemented.
The main advice is to automate somehow the process of setting up a server and displaying changes even for the simplest projects, it’s no more difficult than manual configuration and it pays off many times over.
There are different approaches to this process: python-specific ( fabric , buildout ) or non-specific ( puppet , Chef , sets of shell scripts, etc.).
The fabric approach is a locally executed script that walks ssh to the server and executes commands there. This approach is quite straightforward and easy to debug, and so good ( review on Habré). From the various teams of the fabric, a bike called django-fab-deploy gradually emerged. This is a set of fabric scripts that can configure servers under Debian Lenny or Squeeze, and then with minimal efforts deploy django projects there and manage these projects in the future.
')
With the release of Debian, Squeeze took up django-fab-deploy more seriously, corrected some rough edges and now, I think, it's about time to tell about this project. The project has documentation , there will be a brief retelling with lyrical digressions.
General principles:
- projects are isolated using virtualenv;
- dependencies are managed using requirements pip files;
- all communication with the server is on ssh and must be automated (=> repeatable);
Server part:
- Debian Lenny and Debian Squeeze are supported;
- Python code launch method: apache + mod_wsgi + nginx before them;
- On one VPS / server, you can run as many projects as you like, managed by django-fab-deploy;
- One project can be laid out on several servers (including with different settings).
The main criteria for choosing a deployment method were stability / reliability, and here the bundle apache + mod_wsgi + nginx is currently not equal. I think it’s not difficult to add support for ubunt, but what can I say about this, since there is no support for ubunt right now (I don’t use ubunta myself).
A project would be well kept in a version control system (mercurial is now supported ). There is a half-plug with loading / unpacking tar.gz in case the project is not in VCS; it works, but in the production mode I did not check it (there was no need for it), and there is a chance that some features will appear.
Automate deployment
The following describes the basic way of deploying the server, if for some reason it does not fit, then it is not scary: django-fab-deploy is just a set of scripts; They can be used in parts, or any utility to peep, or send a patch, or just read the article.
It is assumed that we have a clean Debian server (it is possible and not “clean”, more on that later), ssh access via the public key to root is configured.
Please do not use VPS on OpenVZ, because they are limited to VIRT instead of RSS and therefore a lot of software (including Apache and mysql + innodb) under OpenVZ eats much more memory than under xen / kvm (10 times - yes, easily).
Project preparation
Django projects often use some kind of trick with 'local_settings.py', lie in the version control system, have a file with pip dependencies in the repository and a folder with settings for the web server. Django-fab-deploy doesn’t require anything other than that. Further details.
1. Install django-fab-deploy and its dependencies:
pip install django-fab-deploy pip install jinja2 pip install -e git + git: //github.com/bitprophet/fabric.git#egg=Fabric-dev
django-fab-deploy does not work with Fabric 0.9.x, Fabric must be installed from github.
2. Create a file fabfile.py in the project root. This file should contain one or more functions that configure server parameters. Well, because this is a regular fabric script, there may be any other (project-specific) commands. Example:
# my_project/fabfile.py from fab_deploy import * def my_site(): env.hosts = ['my_site@example.com'] env.conf = dict( DB_PASSWORD = 'password', PROCESSES = 2, # , Debian Squeeze; 'lenny' # OS = 'squeeze', # , ( - 'hg') # VCS = 'none', ) update_env() my_site() Pay attention to env.hosts. There is a feature: specify as the user (my_site) the name of your project (it must be the correct name of the variable in python - without dashes and spaces). It doesn't matter if there is no such user, django-fab-deploy will be able to create it and configure ssh access later. In any case, having a separate user for each project is a good idea.
In env.conf, some conventions apply to simplify everything. For example, the database name (DB_NAME) by default is the same as the name of the project instance (INSTANCE_NAME), which, in turn, is the same as the user from env.hosts.
To learn more about what and how you can configure here, see the documentation for fabfile api .
3. Create customization templates for the web server, etc .:
django-fab-deploy config_templates
The config_templates folder will appear in the project root, there are settings templates. Projects are different, and often the settings will require changes - at least fluently read them.
In the settings templates, you can use variables enclosed in {{}} (the template engine here is jinja2). These variables will be replaced with values ​​from the env.conf dictionary when laying out templates on the server.
4. Create a config.server.py in the project root. This file will become the config.py file on the server. Example:
# my_project/config.server.py DEBUG = False DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': '{{ DB_NAME }}', 'USER': '{{ DB_USER }}', 'PASSWORD': '{{ DB_PASSWORD }}', 'HOST': '', 'PORT': '', 'OPTIONS': { "init_command": "SET storage_engine=INNODB" }, } } Create also config.py with local settings for development. Import config.py to settings.py:
# Django settings for my_project project. # ... from config import * # ... The trick is widely known, often config.py also called local_settings.py. This file should be added to ignored in VCS (if VCS is used).
As you can see, config.server.py is also a template (like the files from the config_templates folder), variables from env.conf will also be inserted into it.
5. Create the 'reqs' folder in the project root. An example of such a folder can be obtained by executing the following command from the project root:
django-fab-deploy example_reqs
This folder should contain text files with pip dependencies. One file has a special meaning: reqs / all.txt. This is the main file with dependencies, all packages from there will be installed during deployment. There you can either list all the dependencies, or (better) add other files with dependencies via the “-r” option.
There is also the
django-fab-deploy generate_reqs command, which creates a reqs folder with a single all.txt file, which lists all the python packages installed in the local environment (what pip freeze shows).After steps 1-5 have been completed, the project should look something like this:
my_project
...
config_templates
...
reqs
all.txt
...
fabfile.py
config.py
config.server.py
settings.py
manage.py
If so, then the project is ready.
Server preparation
1. If a non-existent user was specified in env.hosts, then create it and configure ssh access by key, manually or like this (you will need a file with your public key):
fab create_linux_account: "/ home / kmike / .ssh / id_rsa.pub"
Check that ssh works:
ssh my_site@example.com
2. Set up the database. If the database is already configured manually (for example, the project is already running), then nothing needs to be done. django-fab-deploy now can install mysql and create an empty database for a project:
fab mysql_install fab mysql_create_db
mysql_install does nothing if mysql is already installed on the server. If mysql is not installed, then it is set, and the password for the mysql root user is set to env.conf ['DB_PASSWORD'].
mysql_create_db creates an empty database called env.conf ['DB_NAME'] (in our example, this will be the user name from env.hosts).
If you are using not mysql or the user mysql is not root, then it is better to do everything by hand now. Better yet, automate (write a fabric script for this) and send the patch to django-fab-deploy.
3. Everything, the server is ready. You can cross your fingers and run a command from the project root:
fab full_deploy
If everything was done correctly, the site should earn.
This command:
- install the necessary system packages
- will create virtualenv and install dependencies python
- configure apache and nginx
- upload project to server
- will execute syncdb and migrate
Source codes will be in ~ / src / <INSTANCE_NAME>, virtualenv will be placed in ~ / envs / <INSTANCE_NAME>.
django-fab-deploy will disable the “default” sites for Apache and nginx, and will also command Apache ports.conf (Apache will no longer listen to port 80). If other sites were running on the server under Apache, they will become unavailable because of this. If only nginx stuck out, then everything should be fine - django-fab-deploy doesn't do anything tricky with the server.
Server management
Upload changes to the server and apply them:
fab pushAnother example (put the changes on the prod server, while updating the dependencies and performing the migrations):
fab prod push:pip_update,migrateUpdate web server settings:
fab setup_web_serverUpdate janga settings (config.server.py):
fab update_django_configA complete list of commands can be found in the documentation. If you want something more high-level (in the spirit - launched fab redeploy and everything was immediately updated, and the code, settings, and dependencies, and migrations were completed) - you can easily write your fab-command as a wrapper over basic commands. If the push command does too much (it, for example, runs tests by default), then do not hesitate - look at the code and write a more suitable version in your fabfile.py. If you consider any such experiment useful and successful - open the ticket in the bug tracker .
Analogs
The closest analogue to django-fab-deploy is woven . Apparently, also a great thing. Woven is focused on Ubuntu, Debian is supported roughly like Ubuntu in django-fab-deploy: "it seems like it should work with minor changes, but no one knows for sure." We started to do everything at about the same time, at first there was some kind of game in the classes, then they simplified everything. They have advanced far away. On the other hand, in django-fab-deploy is several times smaller than the source code, it is smaller and simpler, and will remain so.
Many people do similar projects. I searched recently for a githab / bitback, and I added 11 more analogs to djangopackages (different degrees of elaboration), you can look here at a comparative tablet: djangopackages.com/grids/g/deployment
Links
Current documentation can always be found here: packages.python.org/django-fab-deploy
Source code and bug tracker repository : bitbucket.org/kmike/django-fab-deploy
Connect: use, correct, write comments, suggestions, how to improve everything, etc.
The description in the article refers to version 0.4 and may be outdated; read, if possible, documentation.
It is not necessary to use django-fab-deploy, you can write your own scripts with 0, use woven, buildout or whatever. But nevertheless, before writing everything with 0, it is better, of course, to do “homework” and see how existing projects are implemented.
The main advice is to automate somehow the process of setting up a server and displaying changes even for the simplest projects, it’s no more difficult than manual configuration and it pays off many times over.
Source: https://habr.com/ru/post/113636/
All Articles