Flask Mega-Tutorial, Part XVII: Deploying Under Linux

(edition 2018)

Miguel grinberg

There

This is the seventeenth part of the Flask Mega-tutorial series, in which I am going to deploy a microblog on a Linux server.

Under the spoiler is a list of all articles in this 2018 series.

Note 1: If you are looking for old versions of this course, this is here .

Note 2: If suddenly you would like to speak in support of my (Miguel) work, or simply do not have the patience to wait for the article for a week, I (Miguel Greenberg) offer the full version of this manual (in English) in the form of an electronic book or video. For more information, visit learn.miguelgrinberg.com .

This chapter is an important milestone in the life of my Microblog application, as it is time to discuss how it can be deployed on a working server for access by real users.

The topic of deployment is extensive, and for this reason it is impossible to cover all possible options within one chapter. This chapter is devoted to the study of traditional hosting options, and I’m going to use a dedicated Linux server running Ubuntu, as well as the widely used Raspberry Pi mini-computer, as research objects. About other options, such as the deployment of clouds and containers, I will discuss in subsequent chapters.

GitHub links for this chapter: Browse , Zip , Diff .

Traditional hosting

When I talk about "traditional hosting", I mean that the application is installed manually or through a script installer on a regular server. The process involves installing the application, its dependencies, and a production scale web server plus setting up the security system.

The first question to ask when deploying your own project is where to find the server. Nowadays, there are many low-cost hosting services. For example, for $ 5 a month, Digital Ocean , Linode, or Amazon Lightsail will lease a virtual Linux server (VPS) for you to conduct deployment experiments. (Linode and Digital Ocean provide their entry-level servers with 1 GB of RAM, while Amazon provides only 512 MB.) If you prefer to practice without spending money, then Vagrant and VirtualBox are two tools that, when combined, allow you to create a virtual server, like a paid one, on your own computer.

As for the choice of operating system, from a technical point of view, this application can be deployed to any of the major operating systems, a list that includes a large number of open source Linux and BSD distributions, as well as commercial OS X and Microsoft Windows ( although OS X is a hybrid version of open-source / commercial, since it is based on Darwin (an open source BSD derivative).

Since OS X and Windows are desktop operating systems that are not optimized to work as servers, I am going to immediately abandon such candidates. The choice between the Linux or BSD operating system is largely based on preferences, so I'm going to choose the most popular of the two, which is Linux. As for Linux distributions, I think it would be fair to make a choice in popularity and continue with Ubuntu.

Ubuntu server

If you are interested in this deployment with me, then you obviously need a server to work with. I will recommend you two options for acquiring a server, one paid and one free. If you are ready to spend some money, you can get an account with Digital Ocean, Linode or Amazon Lightsail and create a virtual server Ubuntu 16.04. You will only need to use the smallest server version, which today, when I write this phrase, costs $ 5 per month for all three suppliers from the above list. Payment is hourly, so if you create a server, play with it for a few hours, and then delete it, you only pay with cents.

The free alternative is based on a virtual machine that you can run on your own computer. To use this option, install Vagrant and VirtualBox on your computer, and then create a file called Vagrantfile to describe the specifications of your virtual machine with the following contents:

Vagrantfile : Vagrant configuration.

Vagrant.configure("2") do |config| config.vm.box = "ubuntu/xenial64" config.vm.network "private_network", ip: "192.168.33.10" config.vm.provider "virtualbox" do |vb| vb.memory = "1024" end end 

This file configures the Ubuntu 16.04 server with 1 GB of RAM, from which you can access the host computer at the IP address 192.168.33.10. To create a server, run the following command:

 $ vagrant up 

In the Vagrant command line documentation , you will learn about other options for managing a virtual server.

Using SSH client

Your server is console, so you will not have a desktop like on your own computer. You will need to connect to your server via an SSH client and work with it via the command line. If you are using Linux or Mac OS X, most likely OpenSSH is already installed. If you are using Microsoft Windows, Cygwin , Git and Windows Subsystem for Linux , they also provide the option to use OpenSSH, so you can install any of these options.

If you are using a third-party virtual server, then when you created the server, you were given an IP address. You can open a terminal session with your new server with the following command:

 $ ssh root@<server-ip-address> 

You will be prompted to enter a password. Depending on the service, the password can be automatically generated and shown to you after the server is created, or perhaps you yourself have given the opportunity to choose your own password.

If you are using a Vagrant VM, you can open a terminal session with the command:

 $ vagrant ssh 

If you are using Windows and Vagrant VM, note that you will need to run the above command from the shell, which can invoke the ssh command from OpenSSH.

Login without password (password-less)

You can skip this section if you are using a Vagrant VM, because your virtual machine is configured correctly to use a non-root account with the name ubuntu without an automatic password from Vagrant.

If you are using a virtual server, it is recommended that you create a regular user account to perform your deployment work and configure this account to log in without using a password (password-less), which may initially seem like a bad decision, but you will see that Not only more convenient, but also safer.

I'm going to create a user account named ubuntu (you can use a different name if you want). To create this account, log in to the root account of your server using the ssh instructions from the previous section, and then enter the following commands to create a user, give it sudo permissions, and finally switch to it:

 $ adduser --gecos "" ubuntu $ usermod -aG sudo ubuntu $ su ubuntu 

Now I am going to configure this new ubuntu account to use public key authentication so that you can log in without entering a password.

Leave the terminal session that you have opened on your server, and start the second terminal on your local computer. If you are using Windows, it should be a terminal from which you have access to the ssh command, so it will probably be a bash or similar prompt, and not a native Windows terminal. In this terminal session, check the contents of the ~ / .ssh directory :

 $ ls ~/.ssh id_rsa id_rsa.pub 

If there are files in the directory list with the name id_rsa and id_rsa.pub , as indicated above, then you already have the key. If you do not have these two files or the ~ / .ssh directory at all, you need to create your own SSH key pair by running the following command, also included in the OpenSSH toolbox:

 $ ssh-keygen 

This application prompts you to enter several values, for which I recommend that you accept the default values ​​by pressing Enter on all prompts. If you know what you are doing and you want to do otherwise, then, as they say, the flag is in your hands.

After executing this command, you should have two files listed above. The id_rsa.pub file is your public key-public key , the file that you provide to third parties for identification. The id_rsa file is your private key-private key that you don’t share with anyone.

Now you need to configure the public key as the authorized host on the server. Print your public key on the screen of the terminal that you opened on your computer:

 $ cat ~/.ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjw....F8Xv4f/0+7WT miguel@miguelspc 

It will be a very long sequence of characters, possibly spanning several lines. You need to copy this data to the clipboard, and then switch back to the terminal on the remote server, where these commands will be issued to store the public key:

 $ echo <paste-your-key-here> >> ~/.ssh/authorized_keys $ chmod 600 ~/.ssh/authorized_keys 

Now the password-less should work. The idea is that ssh on your computer identifies itself on the server, performing a cryptographic operation that requires a secret key. The server then verifies that the operation is valid using the public key.

Now you can log out of your ubuntu session, and then from your root session, and then try to log in directly to your ubuntu account with:

 $ ssh ubuntu@<server-ip-address> 

This time you do not need to enter a password!

Server security

To minimize the risk of server hacking, several steps can be taken to close a number of potential doors through which an attacker can gain access.

The first change that needs to be made is to disable root logins via SSH. Now you have password-less access to your ubuntu account, and you can run admin commands from this account via sudo (on behalf of), so there really is no need to provide a root account. To disable root logins, you need to edit the / etc / ssh / sshd_config file on your server. Perhaps vi and nano (text editors) are already installed on your server and you can use them to edit files (if you are not familiar with any of them, try nano first). You will need to prefix your editor with sudo, because the SSH configuration is not available to regular users (for example, sudo vi /etc/ssh/sshd_config ). You need to change one line in this file:

/ etc / ssh / sshd_config : Disable root logins.

 PermitRootLogin no 

After you finish editing the SSH configuration, you must restart the service for the changes to take effect:

 $ sudo service ssh restart 

The third change to make is to install a firewall. This is software that blocks access to the server on ports that are not explicitly allowed:

 $ sudo apt-get install -y ufw $ sudo ufw allow ssh $ sudo ufw allow http $ sudo ufw allow 443/tcp $ sudo ufw --force enable $ sudo ufw status 

These commands install UFW, a simple firewall, and configure it to allow only external traffic on ports 22 (SSH), 80 (http) and 443 (HTTPS). Any other ports will be closed.

Install Base Dependencies

If you followed my advice and installed the server on Ubuntu 16.04, then you already have a system with full support for Python 3.5, which I am going to use for deployment.

So, we believe that the basic Python interpreter is pre-installed on your server, but there are some additional packages that are likely to be missing, and there are several other packages outside of Python that will be useful in creating a reliable, work-ready deployment. For the database server, I'm going to switch from SQLite to MySQL. postfix is ​​a mail transfer agent that I will use to send email. The supervisor tool will monitor the Flask server process and automatically restart it if it ever crashes or if the server restarts. The nginx server will accept all requests coming from outside and redirect them to the application. Finally, I'm going to use git as a tool to download an application directly from my git repository.

 $ sudo apt-get -y update $ sudo apt-get -y install python3 python3-venv python3-dev $ sudo apt-get -y install mysql-server postfix supervisor nginx git 

These installations are started mostly unattended, but at some point, on the third installation statement, you will be asked to select the root password for the MySQL service, and you will also be asked a few questions regarding the installation of the postfix package, in which you can accept values ​​for default

Note that for this deployment, I prefer not to install Elasticsearch. This service requires a large amount of RAM, so it is viable if you have a large server with more than 2 GB of memory. To avoid problems with low memory on the server, I’ll leave the search functionality. If you have a large enough server, you can download the official .deb package from Elasticsearch and follow their installation instructions to add it to your server. Another important point! The Elasticsearch package available in the repository of the Ubuntu 16.04 package is too old and does not work, you need version 6.x or newer.

It should also be noted that the default installation of postfix is ​​probably not enough to send email in a production environment. To avoid spam and malware, many servers require the sender server to identify itself with security extensions, which means at least you must have a domain name associated with your server. If you want to learn how to completely configure your mail server to pass standard security tests, see the following guides to Digital Ocean:

• Postfix configuration
• Adding an SPF Record
• DKIM Installation and Configuration

Application installation

Now I am going to use git to download the Microblog source code from my GitHub repository. I would recommend you to read git for beginners if you are not familiar with the git version control system.

Note: Three good tutorials for Git learners in Russian:

• Git the simple guide is a simple git guide . But for someone this material may not be enough, because everything is described there very briefly and only the main points.
• Pro Git is an exhaustive Git book that you can buy on Amazon or read online / download for free.
• Git How To is an interactive tour that introduces you to the basics of Git. The tour was created with the understanding that the best way to learn something is to do it yourself.

To download the application to the server, make sure that you are in the home directory of the ubuntu user, and then run:

 $ git clone https://github.com/miguelgrinberg/microblog $ cd microblog $ git checkout v0.17 

This command sequence installs the code on your server and synchronizes it with this chapter. If you maintain your version of this tutorial code in your git repository, you can change the repository URL to your own, in which case you can skip the git checkout command.

Now I need to create a virtual environment and fill it with all the dependencies of the package, which for convenience I saved in the requirements.txt file in Chapter 15 :

 $ python3 -m venv venv $ source venv/bin/activate (venv) $ pip install -r requirements.txt 

In addition to the general required packages in requirements.txt , I'm going to use two more that apply only to this environment, so they are not included in the requirements file. The gunicorn package is a web server for Python applications. The pymysql package contains a MySQL driver that allows SQLAlchemy to work with MySQL databases:

 (venv) $ pip install gunicorn pymysql 

I need to create a .env file with all the necessary environment variables:

/home/ubuntu/microblog/.env : Environment Configuration.

 SECRET_KEY=52cb883e323b48d78a0a36e8e951ba4a MAIL_SERVER=localhost MAIL_PORT=25 DATABASE_URL=mysql+pymysql://microblog:<db-password>@localhost:3306/microblog MS_TRANSLATOR_KEY=<your-translator-key-here> 

This .env file is a bit like the example I showed in chapter 15 , but I used a random string for SECRET_KEY. To generate a random string, I used the following command:

 python3 -c "import uuid; print(uuid.uuid4().hex) 

For the DATABASE_URL variable, I defined the MySQL URL. I will show you how to set up a database in the next section.

I need to set the FLASK_APP environment FLASK_APP to the application entry point for the flask command to work, but this variable must be defined before analyzing the .env file, so it must be set manually. To avoid having to do this every time, I am going to add it to the bottom of ~ / .profile for the ubuntu account, so it will be installed automatically every time I log in:

 $ echo "export FLASK_APP=microblog.py" >> ~/.profile 

If you log out and return, the FLASK_APP value will already be set for you. You can confirm that the value is defined using flask --help . If the auxiliary message shows the translate command added by the application, then the application was found.

And now that the flask command is functional, I can compile the language translations:

 (venv) $ flask translate compile 

MySQL setup

The sqlite database that I used during development is great for simple applications, but when deploying a full-scale web server that can potentially handle multiple queries at the same time, it is better to use a more reliable DBMS. For this reason, I'm going to create a MySQL database, which I will call microblog .

To manage the database server, use the mysql command, which should already be installed on your server:

 $ mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 6 Server version: 5.7.19-0ubuntu0.16.04.1 (Ubuntu) Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> 

Please note that you will need to enter the MySQL root-password that you chose when installing MySQL to access the MySQL command line.

These are the commands that create a new database called microblog and a user with the same name with full access to it:

 mysql> create database microblog character set utf8 collate utf8_bin; mysql> create user 'microblog'@'localhost' identified by '<db-password>'; mysql> grant all privileges on microblog.* to 'microblog'@'localhost'; mysql> flush privileges; mysql> quit; 

You will need to replace <db-password> password of your choice. This will be the password for the microblog database user, so it will be correct that it is different from the password for the root user. The password for the microblog user must match the password included in the DATABASE_URL variable .env file.

If the database configuration is correct, you can run database migrations that create all the tables:

 (venv) $ flask db upgrade 

Before proceeding, make sure that the command above has been completed without error.

Configuring Gunicorn and Supervisor

When you start the server with the flask run command, you use the web server that comes with Flask. This server is useful during development, but it is not a good choice for use on a production server, because it was not built for performance and reliability. Instead of the Flask development server for this deployment, I decided to use gunicorn , which is also a pure Python web server, but unlike Flask, it is a reliable production server that is used by many people, while at the same time it is very easy to use.

To run Microblog under gunicorn, you can use the following command:

 (venv) $ gunicorn -b localhost:8000 -w 4 microblog:app 

The -b option tells gunicorn that port 8000 of the internal network interface is installed to listen to requests. It is generally recommended to run Python web applications without external access, and then have a very fast web server optimized to serve static files that accept all requests from clients. This fast web server will serve static files directly and redirect any requests destined for the application to an internal server. I will show you how to configure nginx as a public server in the next section.

The -w determines how many worker processes will work with gunicorn. The presence of four processes allows an application to process up to four clients simultaneously, which is usually enough for a web application to process a decent number of clients, since not all of them constantly request content. Depending on the amount of RAM available on the server, you may need to adjust the number of processes in order not to run out of memory.

The microblog:app argument tells gunicorn how to download an instance of the application. The name before the colon is the module containing the application, and the name after the colon is the name of this application.

Although gunicorn is very easy to configure, running from the command line is not really a good solution for a production server. I want it to work in the background and be constantly monitored, because if for any reason the server crashes and goes out, I would like to make sure that the new server will automatically start to take its place. In addition, when the computer is restarted, the server should start automatically without the need to log in and start everything on its own. I am going to use the supervisor package that I installed above for this purpose.

The supervisor utility uses configuration files that tell it which programs to monitor and how to restart them if necessary. Configuration files should be stored in /etc/supervisor/conf.d. Here is the configuration file for Microblog, which I am going to call microblog.conf :

/etc/supervisor/conf.d/microblog.conf : Supervisor configuration.

 [program:microblog] command=/home/ubuntu/microblog/venv/bin/gunicorn -b localhost:8000 -w 4 microblog:app directory=/home/ubuntu/microblog user=ubuntu autostart=true autorestart=true stopasgroup=true killasgroup=true 

The command , directory and user settings tell the supervisor how to start the application. autostart and autorestart auto reboot settings due to starting the computer or crashing. The stopasgroup and killasgroup ensure that when the supervisor needs to stop the application in order to restart it, it will also reach all the child processes of the top-level gunicorn process.

After writing this configuration file, you need to restart the supervisor service to import it:

 $ sudo supervisorctl reload 

And likewise, the gunicorn web server must be up and running!

Nginx setup

The gunicorn microblog application server is now running on private port 8000. What I need to do now to give the app to the outside world is to enable my public web server on ports 80 and 443, two ports that I opened on the firewall for processing web traffic applications.

I want this to be a secure deployment, so I'm going to configure port 80 to forward all traffic to port 443, which will be encrypted. So, I'm going to start by creating an SSL certificate. self-signed SSL certificate , , , - , . SSL :

 $ mkdir certs $ openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 \ -keyout certs/key.pem -out certs/cert.pem 

. , SSL, - , . key.pem cert.pem , certs .

- nginx, . nginx /etc/nginx/sites-enabled directory. Nginx , , :

 $ sudo rm /etc/nginx/sites-enabled/default 

nginx Microblog, /etc/nginx/sites-enabled/microblog:

/etc/nginx/sites-enabled/microblog: Nginx configuration.

 server { #   80 (http) listen 80; server_name _; location / { #         URL-,   https return 301 https://$host$request_uri; } } server { #   443 (https) listen 443 ssl; server_name _; #  self-signed SSL- ssl_certificate /home/ubuntu/microblog/certs/cert.pem; ssl_certificate_key /home/ubuntu/microblog/certs/key.pem; #       /var/log access_log /var/log/microblog_access.log; error_log /var/log/microblog_error.log; location / { #      gunicorn proxy_pass http://localhost:8000; proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location /static { #    ,     alias /home/ubuntu/microblog/static; expires 30d; } } 

nginx , , , , , . , nginx .

nginx , :

 $ sudo service nginx reload 

. - IP- ( 192.168.33.10, Vagrant), . , -, .

, . IP- . , , Let's Encrypt SSL. , Flask HTTPS .

Deploying Application Updates

Linux , . git , , , git pull , , .

, , . , , , . , , .

, . , , :

 (venv) $ git pull #    (venv) $ sudo supervisorctl stop microblog #    (venv) $ flask db upgrade #    (venv) $ flask translate compile #   (venv) $ sudo supervisorctl start microblog #    

Raspberry Pi

Raspberry Pi - Linux, , -, 24/7, . Linux, Raspberry Pi. Raspbian , Raspberry Pi Foundation.

Raspberry Pi, Raspbian. raspbian Stretch Lite 2017 , , , , , , .

Raspbian SD-, Raspberry Pi, . Raspbian SD- Windows, Mac OS X Linux Raspberry Pi .

Raspberry Pi , . , SSH, .

Ubuntu, Raspbian Debian, Ubuntu Linux , Raspberry Pi. , . , . SQLite MySQL . nginx gunicorn . , , gunicorn. , , Raspberry Pi.

There