Flask Mega-Tutorial, Part 17: Deploying to Linux (and even to Raspberry Pi!)
This is the seventeenth article in the series, where I describe my experience of writing a Python web application using the Flask mic framework.
The purpose of this guide is to develop a fairly functional microblog application, which I decided to call microblog, in the absence of originality.
Today is a significant event in the life of our application. We are going to present it to the general public and consider various ways to deploy the application and make it available to our users.
')
In this article, I explore the traditional ways of placing an application on the Internet, and therefore focus on using Linux servers, as well as on the very popular Raspberry Pi minicomputer. Cloud services will be covered in the next article.
For a start, we will consider, perhaps, the most well-established type of hosting. We will install our application on a dedicated host running a web server.
Where can we find a host for accommodation? Today there are many offers of inexpensive hosting. The least expensive option is to use VPS (virtual private server) - a virtual machine that looks to us as a dedicated server, but in fact shares physical resources with other hosting users. You can study offers on lowendbox.com if you want to rent a VPS for experiments (note that the link is not referral, which means I won’t get a cent from your click on the link).
Another simple way to get a host to install our application is to install a virtual machine on your own home computer. If you are a Windows or Mac user and would like to gain experience deploying an application on a Linux-managed server without spending a single penny, then this is your option. Install VirtualBox , download the ISO distribution of the Linux distribution of your choice, create a Virtual Machine (VM) and install the OS on it using the previously downloaded image.
Let's talk a little about choosing the OS. I think that the following 4 operating systems should be considered as a platform for hosting a web server:
From a technical point of view, you can use any of these operating systems to host your python application, so how to make a choice?
In my opinion, the choice is not difficult to make. For me, it is important that the platform is open, since a large community of developers and users can cope with the task of maintaining the reliability and safety of the product much better than one company can do by working alone on its closed product. In addition, we would like to have access to a large number of software for installation, because in case of problems, in this case we will have more chances to solve them. Based on the criteria described above, I would prefer to use a server running Linux or BSD to a server running on Windows and OS X.
So the number of options was reduced to two. But the choice between Linux and BSD, as for me, is elementary at all: there is no difference what to use. Both are excellent options for hosting a web server. Depending on the hosting provider, you will most likely be able to choose Linux or BSD, so I just use what is available.
In addition, there are various distributions of Linux and BSD, how to choose from them? This is, for the most part, a matter of personal preference, but if we want to use the most popular distribution, we can divide all distributions into groups according to the basic characteristics, and then choose the most popular of the selected groups:
Surely, my very simplified classification of distributions has left some distributions unattended. I am also sure that many who consider it their duty to remind you that Mac OS X also belongs to the BSD family and should be added to my list. Remember, this is my own classification and I do not expect that everyone around it will agree.
As an exercise in preparing this article, I tested several Linux distributions (the fact is that on my VPS there is no BSD installation option). I tried Fedora, Ubuntu, Debian and CentOS. None of these distributions proved to be easy to install, but of them all, the installation and configuration of CentOS proceeded most smoothly, so that later we will focus on CentOS.
If you have never done the initial server setup before, you may find this exercise very tedious. And you will be right. However, after setting up the server, maintaining its efficiency requires a minimum of effort.
Ready? Then let's get started!
I assume that we have a freshly installed CentOS 6 distribution on a dedicated server or on a VPS. The virtual machine on your PC will also work, but install the OS before continuing.
We will manage the server remotely from our home PC, so we need a tool for registering with the system and running commands. If your PC is running Linux or OS X, then you already have OpenSSH installed. If you are an OS user from Microsoft, then you have a choice of two options:
Cygwin is much more than just SSH, it provides a Linux-like environment inside Windows. Putty, on the other hand, is just an SSH client. Since I plan to cover the very basics, the instructions below will be for users of Linux, OS X and Cygwin. If you decide to use Putty with Windows, then some adaptation of the instructions below may be required.
Let's start with the authorization on our new CentOS 6 server. Open a command prompt on your PC (or Cygwin console if you are using Windows) and run the command:
This command means that you want to log in to the server as the root user, the system administrator. You must replace <your-server> with the IP address or name of your server. Note that you do not have to type the $ character; this is just a command prompt.
You will then be prompted to enter the root password. If it is a dedicated server or VPS, then you entered the password during the installation process. If this is your own virtual machine, then you set a password during installation.
Now that we are logged in, we need to install all the required software, which will most likely include Python and a web server. In addition, we will replace sqlite with a more reliable DBMS - MySQL.
The command used to install software on CentOS is called yum. We can install all required packages with one command:
$ yum install python python-devel httpd httpd-devel mysql-server mysql-devel git gcc sudo
Some of the packages may already be installed, in this case, yum will skip them and install only the missing packages.
We have just installed the following packages:
Now we will set up a separate account under which our application will run. If the reasons for this are not clear, I will explain - the root user has maximum rights, t.ch. we can easily remove or damage important system components as a result of an error. If we work under a limited (user) account, then the important components of the system will be inaccessible to us, and this will protect us from mistakes. And the second, perhaps more significant reason not to use the root account is that if the attacker takes possession of this account, he will gain complete control over the server. We are going to hide the root account as much as possible, I prefer to disable the ability to log in remotely as a root account altogether.
We will call the new account apps, assuming that over time we will have several applications running on this server. Since the applications will run under the web server account, we will add our apps account to the web server group (which was created during the Apache installation), so the web server will have permissions to read and write the files of our application. If you are not familiar with the user rights system in Unix, then a quick guide can be found on Wikipedia .
The command to create user apps on CentOS looks like this:
The -g switch makes the main group of the user being created the apache group, which on CentOS is the apache user group. Further, you will be asked for a password for the user being created.
The adduser team will create a home folder for the new account located at / home / apps. However, if you check the permissions of this folder, we find that it is available only to its owner:
We decided that users of the apache group should have access to this folder, so we will give them maximum rights:
Refer to the chmod documentation for more details. Rights 775 allow full access to the folder to the owner and members of his group, and all others are given all rights except the right to write.
The next step is to allow login under this account from our home PC without using a password. SSH supports another authentication method using a public key . If you are using an OS other than Windows, then it is very likely that you already have keys installed on your system.
Now open the terminal on your PC (Windows users - Cygwin's bash console). To check for keys, run the following command:
If the folder contains id_rsa and id_rsa.pub files, as shown above, you can move on. If these files do not exist, then run the following command:
You will be asked for a few things for which I recommend applying the default values ​​by pressing the Enter key in response to all questions. If you want to do otherwise and at the same time are sure of what you are doing - do it.
After executing this command, you will have the files listed above. The id_rsa.pub file is your public key, you can freely exchange it with other users without being exposed to security threats. The id_rsa file is your private key, which you should never share with anyone.
Now we will add our public key to our server so that it can recognize us from it. In the terminal of our PC, we display the contents of the public key:
Now we need to copy this data to the clipboard, and then switch to CentOS, where to run the following command:
For my combination of server and public key, the following commands are required:
As a result of these commands, your public key will be written to the authorized_keys file on the server. Thus, we made our public key known to OpenSSH on the server. The next task is to secure the .ssh folder and the authorized_keys file inside it:
These commands will change the owner of these files to the apps account, and then make the folder and file available only to the new owner.
Login without a password should already work. Now we will log out of our server’s root account, and log in again, but under the apps user account:
If everything went smoothly, you do not have to enter a password to gain access.
Now we use git to download and install microblog on the server. If you are not familiar with git, I recommend that you familiarize yourself with the git beginner's guide
The application must be located on a git server accessible from our server. I will use my repository on github. You can also use it, or if you prefer, you can clone the repository and make it your own.
To install microblog on our server, the git clone command will be enough for us:
The latest version of the application contains some changes compared to what we saw at the end of the previous article.
First, there are several new files at the topmost level - runp-sqlite.fcgi, runp-mysql.fcgi and killpython. The * .fcgi files are startup scripts used by web servers that work with the FastCGI protocol. The killpython script will restart the application after updates. We will look at this later.
Another notable change is in the config.py file. Until now, we initialized our database like this:
Now we will do it like this:
This simple change will allow us to change the application database by simply setting the environment variable. In the next section, you will see how you can use it.
Another change is in the setup.py file, which was involved in setting up a virtual environment. Now this script also installs the mysql-python package, which is necessary for SQLAlchemy to work with MySQL DBMS. In addition, the Flask-WhooshAlchemy package is now being installed from my own fork, since I did not manage to contact the author of the original package and agree on the inclusion of my corrections in the main distribution.
The sqlite database that we used all this time is perfect for simple applications, but when we move to a full-fledged server that can serve several simultaneous queries, it will be better to use a more reliable DBMS. Therefore, we will configure MySQL for our microblog.
We have already installed MySQL, so all we have to do is create a database and a user with permissions to this database. To manage the database server, we use the mysql console:
And now, on the mysql command line, we will create a database of apps and a user with the same name:
Note that you must specify the password for the apps user in the identified by section. For simplicity, I set the apps password, but in real DB tuning you need to use more secure passwords. Do not confuse user database apps with user system apps.
Now that the database is ready, we can initialize our microblog.
First, create a virtual environment:
Then create a database:
Notice how we set the DATABASE_URL environment variable. Now the application will use MySQL database instead of sqlite.
Next, we compile all messages in our translation database:
And at the end, we give the right to write to the group in two folders that should be accessible to the web server:
The search.db folder is used by our Whoosh full-text search database. The tmp folder should be accessible for writing, for which it will become clear later.
The last thing to do is set up an Apache web server.
We will use the mod_fcigd module to process the FastCGI dialog with our application. Many distributions offer ready-made packages to support mod_fcgid, but unfortunately, CentOS is not one of them, so we just build this module from source.
Here is a list of commands that build and install the current mod_fcgid release:
You can consult the mod_fcgid documentation if you are interested in the details of the above commands.
Having installed this module, we proceed directly to setting up our server. To do this, we need to edit the Apache configuration file. Various Linux distributions have not come to a common opinion about the location and naming of this file, including. you will have to deal with this issue yourself. For example, on CentOS, the configuration file is located at /etc/httpd/conf/httpd.conf.
Here we just add the definition of our server. At the end of the file add the following lines (do this as root administrator, since this file is not writeable for ordinary users):
The FcgidIPCDir parameter points to the folder where the socket files will be created. I found that by default they are created in a folder in which the apache user does not have write access, so I decided to place these files in / tmp.
Then we specify AddHandler to tell apache that all files with the .fcgi extension should be interpreted as FastCGI files and run through the mod_fcgid module that we just installed. Remember the new file runp-mysql.fcgi which is located in our root directory? This file uses the python flup module as an adapter so that our application can interact with the FastCGI protocol. Let's take a look at this file:
Apache will execute this file to run our application. Notice how we put the name of the MySQL database in the environment so that Apache can recognize it.
The section defines the host that will serve the web server. The designation *: 80 means that all requests received by the server on any host on port 80 will be serviced by this virtual server. Several sections can be defined in the configuration file, each of which defines a different server. In order to distinguish servers from each other, you can use different IP addresses, domains / subdomains, or ports.
The definition of our virtual host is quite simple. The DocumentRoot operator tells Apache where to look for static files. All file requests will be served from this folder, so for example, when the browser requests the /favicon.ico file, Apache will search for it just here to display a small icon next to the address entry field in the browser. However, all static files used in our application will have the / static prefix, so in order for Apache not to look for another static folder, we use the Alias ​​operator, which means that all requests starting with / static must fall into our folder with statics. Finally, the ScriptAlias ​​operator tells Apache that when receiving a request starting with / (actually, all requests except static requests) need to call the script specified by the second argument, and this is our .fcgi script.
The Apache configuration file supports many more options than I mentioned. I recommend that you familiarize yourself with the Apache documentation in order to decide what other options it makes sense to use for your server.
To activate the changes, we need to restart the server from under the root account:
Now, by going to http: // <your-server> in your browser, you should see the pages of our microblog.
The last point that we will consider is rolling out updates of our application.
Suppose we have deployed the application and it has successfully worked for some time on our server. And now it's time to roll out the update, which should fix some errors found or simply add new features.
We use git and our own toolkit for this. If we are authorized on the server under the apps account, we can upgrade to the latest release in the repository using the commands:
The git pull command will download all new and updated files from our git server. Then, we will update our database and recompile the translation files if they have changed. And it's all!
Now that all updates have been received, it is necessary to restart the FastCGI processes, and this is somewhat inconvenient, since we are logged in as the user of the apps, the non-privileged user, while the FastCGI processes belong to the apache user.
And in this case sudo is useful to us. The sudo command allows users to run applications on behalf of another user. We would not like to give our user apps too many rights, so we only give him the right to send FastCGI stop signals to processes running by the apache user.
The killpython script from our microblog folder does just that:
The hitch is that if we run this script on behalf of the user apps, we will not have the necessary privileges to stop the processes owned by the apache user. To allow the apps user to interrupt these processes, we need to make small changes to the / etc / sudoers file:
Note that you should make these changes as root, because / etc / sudoers is a system file.
The first hard-to-read command gives apps the user the right to run killpython as an apache user without having to enter a password. The second line allows the apps user to use the sudo command by calling it from a script, and not just from the console. You can refer to the sudoers file guide for detailed information on the sudo configuration konfig syntax.
Now that we’ve logged in as the apps user, we can interrupt the python processes by running the command:
If the FastCGI processes are interrupted, Apache will restart them by receiving the following request to the microblog.
To simplify the update of our server a little more, we could create a client script that would combine the update and restart processes. The script itself would look something like this:
If you save this command in a script, then you can roll out the update to the server with one command!
I have not considered a few normal operations that are recommended for use to ensure even more server security in the “hostile environment of the Internet.” These are operations such as:
I will leave these topics as an exercise for an interested reader, as they have a mediocre relationship to the topic of this article.
Raspberry Pi is a revolutionary miniature Linux-based computer that costs about $ 35. It has a very low power consumption, which makes it an excellent device for hosting home web applications that are available online 24/7 without having to keep a large PC on constantly.
There are several distributions designed for the Raspberry Pi. We will install our application on Raspbian , the official distribution.
By the way, note that Raspbian is an offshoot of the Debian distribution, which means that the instructions below will apply (possibly with minimal adjustments) to Debian / Ubuntu-based servers.
Now we will repeat the same steps that we went through for CentOS to configure the RPi server.
Raspberry Pi is a full-fledged computer. You can connect an HDMI monitor, keyboard, mouse and work directly on it to configure the server software. Sort of like we did on CentOS (and indeed on any Linux distribution), you can simply connect the Raspberry Pi to the network, and then connect to it using ssh. See the “Client Side Configuration” subsection above to configure ssh on your PC.
Notice that I did not touch the installation and configuration of the Raspberry Pi itself, since there are a lot of articles written about it. I assume that the Raspberry Pi is already running and connected to the network.
SincePi still represents a machine with limited capacity, we will not install a full-fledged Apache / MySQL stack, as we did on CentOS. Instead, we will take the easy approach. As a web server we use Lighttpd , a small web server with a very good and efficient implementation of FastCGI. As a DBMS, we will leave sqlite.
For CentOS, we used yum to install packages. On the distributions of the same Debian family, the package manager apt-get is used:
The update command downloads and updates the list of packages to your Pi. The upgrade command upgrades all installed packages to their latest versions. It is recommended to run these commands periodically in order to keep your software up to date. And finally, the install command installs the packages we need.
We decided that our Raspberry Pi server would be located on a trusted network. Therefore, security is not our main priority in these conditions, and we will not create a separate user, but instead use the default account - pi.
If you plan to keep the Raspberry Pi connected to the Internet, then you should use the approach we used for CentOS and create a separate limited account, so that if this account is hacked, the damage caused will be limited.
Establishing public keys is also not a very important procedure if the application runs on a local network, but if you need this functionality, the procedure will be completely similar to the one we used on CentOS.
The application is installed using git:
Since we plan to use sqlite on this host, we can initialize microblog the way we did in previous articles for our development server:
Note that setup.py will try to install support for MySQL and this attempt will fail, but this is not terrible, because we do not use MySQL.
All files are owned by user pi. But as we saw in the case of CentOS, the web server will be launched on behalf of another user. Since Security is not a necessity on this server, we will proceed more simply:
This command will give write rights (and indeed all rights) to all users. Again, this is a very bad idea to do so on an open server, but for a server under your complete control, this can eliminate confusion about access rights.
I would like to clarify that the Raspberry Pi is perfectly compatible with the configuration based on groups and access rights, like the one we used for the server running CentOS. We chose a simplified configuration scheme, since we will use this server in a fully controlled environment.
Lighttpd has built-in support for FastCGI, which means we don’t have to install a separate module for this purpose.
All we need to do is add our website to the configuration file located at /etc/lighttpd/lighttpd.conf:
And after that, our application should already be available online at http: // <IP address of Raspberry Pi>.
However, you need to note a few points in the presented configuration, as well as a few things that initially do not work and require some changes on the application side.
The fastcgi.server expression defines the behavior of our FastCGI server, available at / microblog. The reason that we did not place the application at the address / is simple, maybe later we will run more than one application, in which case the placement of all the addresses of our application in / microblog will play the role of the namespace.
Inside the definition of FastCGI, we set the location of the socket file in the / tmp folder and the path to our runp-sqlite.fcgi file, which Lighttpd will launch to start FastCGI processes. The check-local option tells Lighttpd to send FastCGI requests to the server, even if the requested path matches a file on disk. max-procs limits the number of FastCGI processes to 1, which is quite enough for a small server, and also helps to avoid potential problems with several simultaneous write requests to the sqlite database.
The alias.url section reflects static queries for the local path to the corresponding file ..
The runp-sqlite.fcgi script is very similar to the one we used on CentOS, only without overwriting the database settings:
Testing this setup will indicate several problems with our application.
Firstly, our javascript and css files are not loaded. The problem is easily solved by looking at the source of the page on the login page. In the base.html template, these files are given in absolute paths that begin with / static, however, we added the prefix / microblog to our addresses. For example, here’s a link to a CSS file:
At that time, when in fact we would like to see /microblog/static/bootstrap.min.css here, and, of course, we cannot simply add / microblog to the base.html template because it will cause our application to fail on development server running on our PC. The solution is to generate these addresses automatically using url_for:
After the update, all js and css files will be loaded correctly.
But an attempt to log in will reveal a new problem. Immediately after logging in, we get error 500. Fortunately for us, we have logging enabled, so a quick examination of /home/pi/microblog/tmp/microblog.log leads us to the following error:
What is this?We do not run multiple threads, but it is obvious that our application still uses multiple threads. The only difference between the server for development and this is that here we use the flup for the FastCGI server. Looking into the source code of this module, we learn that by default, flup starts a multi-threaded server.
A possible solution would be to use a single FastCGI server thread provided by flup. However, this can hurt performance in the case of multiple concurrent connections. Another more promising way to solve this problem is to allow multithreading in sqlite (which seems to be fully supported ). Multithreading can be enabled in SQLAlchemy by setting the check_same_thread parameter to False in our config.py configuration file:
With this change, we allowed to run many threads to make changes to the database, while the sqlite library assumes synchronization of multiple access.
And finally, we run our app on the Raspberry Pi!
To install updates, we can simply log in to the Raspberry Pi and call the commands that update the source code, database and translation files:
If you have configured a passwordless entry to your Pi, then you can easily write a script that executes all of this in one command.
The updated application is available, as always, on github . Or you can download it as a zip archive by the link:
Download microblog 0.17 .
In the next article, we will look at deployment on a cloud service, as an alternative to the traditional approach that we reviewed today.
See you later!
Miguel
The purpose of this guide is to develop a fairly functional microblog application, which I decided to call microblog, in the absence of originality.
Table of contents
Part 1: Hello, World!
Part 2: Templates
Part 3: Forms
Part 4: Database
Part 5: User Login
Part 6: Profile Page and Avatars
Part 7: Unit Testing
Part 8: Subscribers, Contacts and Friends
Part 9: Pagination
Part 10: Full Text Search
Part 11: Email Support
Part 12: Reconstruction
Part 13: Date and Time
Part 14: I18n and L10n
Part 15: Ajax
Part 16: Debugging, Testing, and Profiling
Part 17: Deploying to Linux (and even to Raspberry Pi!) (This article)
Part 18: Deploying to Heroku Cloud
Part 2: Templates
Part 3: Forms
Part 4: Database
Part 5: User Login
Part 6: Profile Page and Avatars
Part 7: Unit Testing
Part 8: Subscribers, Contacts and Friends
Part 9: Pagination
Part 10: Full Text Search
Part 11: Email Support
Part 12: Reconstruction
Part 13: Date and Time
Part 14: I18n and L10n
Part 15: Ajax
Part 16: Debugging, Testing, and Profiling
Part 17: Deploying to Linux (and even to Raspberry Pi!) (This article)
Part 18: Deploying to Heroku Cloud
Today is a significant event in the life of our application. We are going to present it to the general public and consider various ways to deploy the application and make it available to our users.
')
In this article, I explore the traditional ways of placing an application on the Internet, and therefore focus on using Linux servers, as well as on the very popular Raspberry Pi minicomputer. Cloud services will be covered in the next article.
For a start, we will consider, perhaps, the most well-established type of hosting. We will install our application on a dedicated host running a web server.
Where can we find a host for accommodation? Today there are many offers of inexpensive hosting. The least expensive option is to use VPS (virtual private server) - a virtual machine that looks to us as a dedicated server, but in fact shares physical resources with other hosting users. You can study offers on lowendbox.com if you want to rent a VPS for experiments (note that the link is not referral, which means I won’t get a cent from your click on the link).
Another simple way to get a host to install our application is to install a virtual machine on your own home computer. If you are a Windows or Mac user and would like to gain experience deploying an application on a Linux-managed server without spending a single penny, then this is your option. Install VirtualBox , download the ISO distribution of the Linux distribution of your choice, create a Virtual Machine (VM) and install the OS on it using the previously downloaded image.
Let's talk a little about choosing the OS. I think that the following 4 operating systems should be considered as a platform for hosting a web server:
- Linux
- BSD
- OS X
- Windows
From a technical point of view, you can use any of these operating systems to host your python application, so how to make a choice?
In my opinion, the choice is not difficult to make. For me, it is important that the platform is open, since a large community of developers and users can cope with the task of maintaining the reliability and safety of the product much better than one company can do by working alone on its closed product. In addition, we would like to have access to a large number of software for installation, because in case of problems, in this case we will have more chances to solve them. Based on the criteria described above, I would prefer to use a server running Linux or BSD to a server running on Windows and OS X.
So the number of options was reduced to two. But the choice between Linux and BSD, as for me, is elementary at all: there is no difference what to use. Both are excellent options for hosting a web server. Depending on the hosting provider, you will most likely be able to choose Linux or BSD, so I just use what is available.
In addition, there are various distributions of Linux and BSD, how to choose from them? This is, for the most part, a matter of personal preference, but if we want to use the most popular distribution, we can divide all distributions into groups according to the basic characteristics, and then choose the most popular of the selected groups:
- RPM based (RedHat, CentOS, Fedora)
- Debian based (Debian, Ubuntu, Mint)
- BSD based (FreeBSD, NetBSD, OpenBSD)
- other
Surely, my very simplified classification of distributions has left some distributions unattended. I am also sure that many who consider it their duty to remind you that Mac OS X also belongs to the BSD family and should be added to my list. Remember, this is my own classification and I do not expect that everyone around it will agree.
As an exercise in preparing this article, I tested several Linux distributions (the fact is that on my VPS there is no BSD installation option). I tried Fedora, Ubuntu, Debian and CentOS. None of these distributions proved to be easy to install, but of them all, the installation and configuration of CentOS proceeded most smoothly, so that later we will focus on CentOS.
If you have never done the initial server setup before, you may find this exercise very tedious. And you will be right. However, after setting up the server, maintaining its efficiency requires a minimum of effort.
Ready? Then let's get started!
Hosting on CentOS 6
I assume that we have a freshly installed CentOS 6 distribution on a dedicated server or on a VPS. The virtual machine on your PC will also work, but install the OS before continuing.
Client side customization
We will manage the server remotely from our home PC, so we need a tool for registering with the system and running commands. If your PC is running Linux or OS X, then you already have OpenSSH installed. If you are an OS user from Microsoft, then you have a choice of two options:
Cygwin is much more than just SSH, it provides a Linux-like environment inside Windows. Putty, on the other hand, is just an SSH client. Since I plan to cover the very basics, the instructions below will be for users of Linux, OS X and Cygwin. If you decide to use Putty with Windows, then some adaptation of the instructions below may be required.
Let's start with the authorization on our new CentOS 6 server. Open a command prompt on your PC (or Cygwin console if you are using Windows) and run the command:
$ ssh root@<your-server> This command means that you want to log in to the server as the root user, the system administrator. You must replace <your-server> with the IP address or name of your server. Note that you do not have to type the $ character; this is just a command prompt.
You will then be prompted to enter the root password. If it is a dedicated server or VPS, then you entered the password during the installation process. If this is your own virtual machine, then you set a password during installation.
Installing software packages
Now that we are logged in, we need to install all the required software, which will most likely include Python and a web server. In addition, we will replace sqlite with a more reliable DBMS - MySQL.
The command used to install software on CentOS is called yum. We can install all required packages with one command:
$ yum install python python-devel httpd httpd-devel mysql-server mysql-devel git gcc sudo
Some of the packages may already be installed, in this case, yum will skip them and install only the missing packages.
We have just installed the following packages:
- python and python-devel: Python interpreter and its developer tools (header files and static libraries)
- httpd and httpd-devel: Apache web server and its developer tools
- mysql-server and mysql-devel: MySQL database server and its developer tools
- git: distributed file version control system (we will use it to download and update the application)
- gcc: C / C ++ compiler (needed to compile Python extensions)
- sudo: a tool to run commands on behalf of another user.
Creating a separate user for our application
Now we will set up a separate account under which our application will run. If the reasons for this are not clear, I will explain - the root user has maximum rights, t.ch. we can easily remove or damage important system components as a result of an error. If we work under a limited (user) account, then the important components of the system will be inaccessible to us, and this will protect us from mistakes. And the second, perhaps more significant reason not to use the root account is that if the attacker takes possession of this account, he will gain complete control over the server. We are going to hide the root account as much as possible, I prefer to disable the ability to log in remotely as a root account altogether.
We will call the new account apps, assuming that over time we will have several applications running on this server. Since the applications will run under the web server account, we will add our apps account to the web server group (which was created during the Apache installation), so the web server will have permissions to read and write the files of our application. If you are not familiar with the user rights system in Unix, then a quick guide can be found on Wikipedia .
The command to create user apps on CentOS looks like this:
$ adduser -g apache apps The -g switch makes the main group of the user being created the apache group, which on CentOS is the apache user group. Further, you will be asked for a password for the user being created.
The adduser team will create a home folder for the new account located at / home / apps. However, if you check the permissions of this folder, we find that it is available only to its owner:
$ ls /home -l total 7 drwx------ 2 apps apache 4096 Apr 7 11:46 apps We decided that users of the apache group should have access to this folder, so we will give them maximum rights:
$ chmod 775 /home/apps Refer to the chmod documentation for more details. Rights 775 allow full access to the folder to the owner and members of his group, and all others are given all rights except the right to write.
Login without password
The next step is to allow login under this account from our home PC without using a password. SSH supports another authentication method using a public key . If you are using an OS other than Windows, then it is very likely that you already have keys installed on your system.
Now open the terminal on your PC (Windows users - Cygwin's bash console). To check for keys, run the following command:
$ ls ~/.ssh id_rsa id_rsa.pub If the folder contains id_rsa and id_rsa.pub files, as shown above, you can move on. If these files do not exist, then run the following command:
$ keygen -t rsa You will be asked for a few things for which I recommend applying the default values ​​by pressing the Enter key in response to all questions. If you want to do otherwise and at the same time are sure of what you are doing - do it.
After executing this command, you will have the files listed above. The id_rsa.pub file is your public key, you can freely exchange it with other users without being exposed to security threats. The id_rsa file is your private key, which you should never share with anyone.
Now we will add our public key to our server so that it can recognize us from it. In the terminal of our PC, we display the contents of the public key:
$ cat ~/.ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjwaK4JVuY6PZAr8HocCIOszrLIzzCjO0Xlt9zkFNKvVpP1B92u3JvwwiagqR+k0kHih2SmYnycmXjAcE60tvu+sIDA/7tEJZh4kO4nUYM5PJ17E+qTqUleBXQM74eITydq/USkOqc5p++qUUgA60gUUuNum3igbZiNi71zK4m8g/IDywWYk+5vzNt2i7Sm8NEuauy/xWgnWhCBXZ/tXfkgWgC/4HzpmsfO+nniNh8VgTZp8Q+y+4psSE+p14qUg7KdDbf0Wo/D35wDkMvto96bIT8RF0np9dTkFj8TgNW8inP+6MC+4vCd8F/NpESCVt8hRlBVERMF8Xv4f/0+7WT miguel@miguelspc Now we need to copy this data to the clipboard, and then switch to CentOS, where to run the following command:
$ mkdir /home/apps/.ssh $ echo <paste-your-key-here> > /home/apps/.ssh/authorized_keys For my combination of server and public key, the following commands are required:
$ mkdir /home/apps/.ssh $ echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjwaK4JVuY6PZAr8HocCIOszrLIzzCjO0Xlt9zkFNKvVpP1B92u3JvwwiagqR+k0kHih2SmYnycmXjAcE60tvu+sIDA/7tEJZh4kO4nUYM5PJ17E+qTqUleBXQM74eITydq/USkOqc5p++qUUgA60gUUuNum3igbZiNi71zK4m8g/IDywWYk+5vzNt2i7Sm8NEuauy/xWgnWhCBXZ/tXfkgWgC/4HzpmsfO+nniNh8VgTZp8Q+y+4psSE+p14qUg7KdDbf0Wo/D35wDkMvto96bIT8RF0np9dTkFj8TgNW8inP+6MC+4vCd8F/NpESCVt8hRlBVERMF8Xv4f/0+7WT miguel@miguelspc > /home/apps/.ssh/authorized_keys As a result of these commands, your public key will be written to the authorized_keys file on the server. Thus, we made our public key known to OpenSSH on the server. The next task is to secure the .ssh folder and the authorized_keys file inside it:
$ chown -R apps:apache /home/apps/.ssh $ chmod 700 /home/apps/.ssh $ chmod 600 /home/apps/.ssh/authorized_keys These commands will change the owner of these files to the apps account, and then make the folder and file available only to the new owner.
Login without a password should already work. Now we will log out of our server’s root account, and log in again, but under the apps user account:
$ ssh apps@<your-server> If everything went smoothly, you do not have to enter a password to gain access.
Application installation
Now we use git to download and install microblog on the server. If you are not familiar with git, I recommend that you familiarize yourself with the git beginner's guide
The application must be located on a git server accessible from our server. I will use my repository on github. You can also use it, or if you prefer, you can clone the repository and make it your own.
To install microblog on our server, the git clone command will be enough for us:
$ git clone git://github.com/miguelgrinberg/microblog.git $ cd microblog The latest version of the application contains some changes compared to what we saw at the end of the previous article.
First, there are several new files at the topmost level - runp-sqlite.fcgi, runp-mysql.fcgi and killpython. The * .fcgi files are startup scripts used by web servers that work with the FastCGI protocol. The killpython script will restart the application after updates. We will look at this later.
Another notable change is in the config.py file. Until now, we initialized our database like this:
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db') Now we will do it like this:
if os.environ.get('DATABASE_URL') is None: SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db') else: SQLALCHEMY_DATABASE_URI = os.environ['DATABASE_URL'] This simple change will allow us to change the application database by simply setting the environment variable. In the next section, you will see how you can use it.
Another change is in the setup.py file, which was involved in setting up a virtual environment. Now this script also installs the mysql-python package, which is necessary for SQLAlchemy to work with MySQL DBMS. In addition, the Flask-WhooshAlchemy package is now being installed from my own fork, since I did not manage to contact the author of the original package and agree on the inclusion of my corrections in the main distribution.
MySQL setup
The sqlite database that we used all this time is perfect for simple applications, but when we move to a full-fledged server that can serve several simultaneous queries, it will be better to use a more reliable DBMS. Therefore, we will configure MySQL for our microblog.
We have already installed MySQL, so all we have to do is create a database and a user with permissions to this database. To manage the database server, we use the mysql console:
$ mysql -u root -p Enter password: (enter the mysql root password, or empty if one is not defined) Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 11 Server version: 5.1.67 Source distribution Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> And now, on the mysql command line, we will create a database of apps and a user with the same name:
mysql> create database apps character set utf8 collate utf8_bin; mysql> create user 'apps'@'localhost' identified by 'apps'; mysql> grant all privileges on apps.* to 'apps'@'localhost'; mysql> flush privileges; mysql> quit; Note that you must specify the password for the apps user in the identified by section. For simplicity, I set the apps password, but in real DB tuning you need to use more secure passwords. Do not confuse user database apps with user system apps.
Application Initialization
Now that the database is ready, we can initialize our microblog.
First, create a virtual environment:
$ ./setup.py Then create a database:
$ DATABASE_URL=mysql://apps:apps@localhost/apps ./db_create.py Notice how we set the DATABASE_URL environment variable. Now the application will use MySQL database instead of sqlite.
Next, we compile all messages in our translation database:
$ ./tr_compile.py And at the end, we give the right to write to the group in two folders that should be accessible to the web server:
$ chmod -R g+w search.db tmp The search.db folder is used by our Whoosh full-text search database. The tmp folder should be accessible for writing, for which it will become clear later.
Apache setup
The last thing to do is set up an Apache web server.
We will use the mod_fcigd module to process the FastCGI dialog with our application. Many distributions offer ready-made packages to support mod_fcgid, but unfortunately, CentOS is not one of them, so we just build this module from source.
Here is a list of commands that build and install the current mod_fcgid release:
$ wget http://mirror.metrocast.net/apache//httpd/mod_fcgid/mod_fcgid-2.3.7.tar.gz $ tar xvzf mod_fcgid-2.3.7.tar.gz $ cd mod_fcgid-2.3.7 $ APXS=/usr/sbin/apxs ./configure.apxs $ su (enter root password) $ make install You can consult the mod_fcgid documentation if you are interested in the details of the above commands.
Having installed this module, we proceed directly to setting up our server. To do this, we need to edit the Apache configuration file. Various Linux distributions have not come to a common opinion about the location and naming of this file, including. you will have to deal with this issue yourself. For example, on CentOS, the configuration file is located at /etc/httpd/conf/httpd.conf.
Here we just add the definition of our server. At the end of the file add the following lines (do this as root administrator, since this file is not writeable for ordinary users):
FcgidIPCDir /tmp AddHandler fcgid-script .fcgi <VirtualHost *:80> DocumentRoot /home/apps/microblog/app/static Alias /static /home/apps/microblog/app/static ScriptAlias / /home/apps/microblog/runp-mysql.fcgi/ </VirtualHost> The FcgidIPCDir parameter points to the folder where the socket files will be created. I found that by default they are created in a folder in which the apache user does not have write access, so I decided to place these files in / tmp.
Then we specify AddHandler to tell apache that all files with the .fcgi extension should be interpreted as FastCGI files and run through the mod_fcgid module that we just installed. Remember the new file runp-mysql.fcgi which is located in our root directory? This file uses the python flup module as an adapter so that our application can interact with the FastCGI protocol. Let's take a look at this file:
#!flask/bin/python import os os.environ['DATABASE_URL'] = 'mysql://apps:apps@localhost/apps' from flup.server.fcgi import WSGIServer from app import app if __name__ == '__main__': WSGIServer(app).run() Apache will execute this file to run our application. Notice how we put the name of the MySQL database in the environment so that Apache can recognize it.
The section defines the host that will serve the web server. The designation *: 80 means that all requests received by the server on any host on port 80 will be serviced by this virtual server. Several sections can be defined in the configuration file, each of which defines a different server. In order to distinguish servers from each other, you can use different IP addresses, domains / subdomains, or ports.
The definition of our virtual host is quite simple. The DocumentRoot operator tells Apache where to look for static files. All file requests will be served from this folder, so for example, when the browser requests the /favicon.ico file, Apache will search for it just here to display a small icon next to the address entry field in the browser. However, all static files used in our application will have the / static prefix, so in order for Apache not to look for another static folder, we use the Alias ​​operator, which means that all requests starting with / static must fall into our folder with statics. Finally, the ScriptAlias ​​operator tells Apache that when receiving a request starting with / (actually, all requests except static requests) need to call the script specified by the second argument, and this is our .fcgi script.
The Apache configuration file supports many more options than I mentioned. I recommend that you familiarize yourself with the Apache documentation in order to decide what other options it makes sense to use for your server.
To activate the changes, we need to restart the server from under the root account:
# service httpd restart Now, by going to http: // <your-server> in your browser, you should see the pages of our microblog.
Install application updates
The last point that we will consider is rolling out updates of our application.
Suppose we have deployed the application and it has successfully worked for some time on our server. And now it's time to roll out the update, which should fix some errors found or simply add new features.
We use git and our own toolkit for this. If we are authorized on the server under the apps account, we can upgrade to the latest release in the repository using the commands:
$ cd microblog $ git pull $ DATABASE_URL=mysql://apps:apps@localhost/apps ./db_upgrade.py $ ./tr_compile.py The git pull command will download all new and updated files from our git server. Then, we will update our database and recompile the translation files if they have changed. And it's all!
Now that all updates have been received, it is necessary to restart the FastCGI processes, and this is somewhat inconvenient, since we are logged in as the user of the apps, the non-privileged user, while the FastCGI processes belong to the apache user.
And in this case sudo is useful to us. The sudo command allows users to run applications on behalf of another user. We would not like to give our user apps too many rights, so we only give him the right to send FastCGI stop signals to processes running by the apache user.
The killpython script from our microblog folder does just that:
killall /home/apps/microblog/flask/bin/python The hitch is that if we run this script on behalf of the user apps, we will not have the necessary privileges to stop the processes owned by the apache user. To allow the apps user to interrupt these processes, we need to make small changes to the / etc / sudoers file:
apps ALL=(apache) NOPASSWD:/home/apps/microblog/killpython Defaults: apps !requiretty Note that you should make these changes as root, because / etc / sudoers is a system file.
The first hard-to-read command gives apps the user the right to run killpython as an apache user without having to enter a password. The second line allows the apps user to use the sudo command by calling it from a script, and not just from the console. You can refer to the sudoers file guide for detailed information on the sudo configuration konfig syntax.
Now that we’ve logged in as the apps user, we can interrupt the python processes by running the command:
$ sudo -u apache ./killpython If the FastCGI processes are interrupted, Apache will restart them by receiving the following request to the microblog.
To simplify the update of our server a little more, we could create a client script that would combine the update and restart processes. The script itself would look something like this:
ssh apps@<your-server> "cd microblog;git pull;sudo -u apache ./killpython;DATABASE_URL=mysql://apps:apps@localhost/apps ./db_upgrade.py;./tr_compile.py" If you save this command in a script, then you can roll out the update to the server with one command!
What else can you add
I have not considered a few normal operations that are recommended for use to ensure even more server security in the “hostile environment of the Internet.” These are operations such as:
- Prevent remote root login
- Disable all unused services like FTP, CIFS, etc.
- Firewall configuration.
- Timely installation of security updates for server software.
- Some others.
I will leave these topics as an exercise for an interested reader, as they have a mediocre relationship to the topic of this article.
Hosting on Raspberry Pi
Raspberry Pi is a revolutionary miniature Linux-based computer that costs about $ 35. It has a very low power consumption, which makes it an excellent device for hosting home web applications that are available online 24/7 without having to keep a large PC on constantly.
There are several distributions designed for the Raspberry Pi. We will install our application on Raspbian , the official distribution.
By the way, note that Raspbian is an offshoot of the Debian distribution, which means that the instructions below will apply (possibly with minimal adjustments) to Debian / Ubuntu-based servers.
Now we will repeat the same steps that we went through for CentOS to configure the RPi server.
Client side customization
Raspberry Pi is a full-fledged computer. You can connect an HDMI monitor, keyboard, mouse and work directly on it to configure the server software. Sort of like we did on CentOS (and indeed on any Linux distribution), you can simply connect the Raspberry Pi to the network, and then connect to it using ssh. See the “Client Side Configuration” subsection above to configure ssh on your PC.
Notice that I did not touch the installation and configuration of the Raspberry Pi itself, since there are a lot of articles written about it. I assume that the Raspberry Pi is already running and connected to the network.
Installing software packages
SincePi still represents a machine with limited capacity, we will not install a full-fledged Apache / MySQL stack, as we did on CentOS. Instead, we will take the easy approach. As a web server we use Lighttpd , a small web server with a very good and efficient implementation of FastCGI. As a DBMS, we will leave sqlite.
For CentOS, we used yum to install packages. On the distributions of the same Debian family, the package manager apt-get is used:
$ sudo apt-get update $ sudo apt-get upgrade $ sudo apt-get install python python-dev lighttpd git The update command downloads and updates the list of packages to your Pi. The upgrade command upgrades all installed packages to their latest versions. It is recommended to run these commands periodically in order to keep your software up to date. And finally, the install command installs the packages we need.
Creating a separate user for our application
We decided that our Raspberry Pi server would be located on a trusted network. Therefore, security is not our main priority in these conditions, and we will not create a separate user, but instead use the default account - pi.
If you plan to keep the Raspberry Pi connected to the Internet, then you should use the approach we used for CentOS and create a separate limited account, so that if this account is hacked, the damage caused will be limited.
Login without password
Establishing public keys is also not a very important procedure if the application runs on a local network, but if you need this functionality, the procedure will be completely similar to the one we used on CentOS.
Application installation
The application is installed using git:
$ git clone git://github.com/miguelgrinberg/microblog.git $ cd microblog Application Initialization
Since we plan to use sqlite on this host, we can initialize microblog the way we did in previous articles for our development server:
$ ./setup.py $ ./db_create.py $ ./tr_compile.py Note that setup.py will try to install support for MySQL and this attempt will fail, but this is not terrible, because we do not use MySQL.
All files are owned by user pi. But as we saw in the case of CentOS, the web server will be launched on behalf of another user. Since Security is not a necessity on this server, we will proceed more simply:
$ chmod -R 777 * This command will give write rights (and indeed all rights) to all users. Again, this is a very bad idea to do so on an open server, but for a server under your complete control, this can eliminate confusion about access rights.
I would like to clarify that the Raspberry Pi is perfectly compatible with the configuration based on groups and access rights, like the one we used for the server running CentOS. We chose a simplified configuration scheme, since we will use this server in a fully controlled environment.
Lighttpd setup
Lighttpd has built-in support for FastCGI, which means we don’t have to install a separate module for this purpose.
All we need to do is add our website to the configuration file located at /etc/lighttpd/lighttpd.conf:
fastcgi.server = ("/microblog" => (( "socket" => "/tmp/microblog-fcgi.sock", "bin-path" => "/home/pi/microblog/runp-sqlite.fcgi", "check-local" => "disable", "max-procs" => 1 )) ) alias.url = ( "/microblog/static/" => "/home/pi/microblog/app/static/", ) And after that, our application should already be available online at http: // <IP address of Raspberry Pi>.
However, you need to note a few points in the presented configuration, as well as a few things that initially do not work and require some changes on the application side.
The fastcgi.server expression defines the behavior of our FastCGI server, available at / microblog. The reason that we did not place the application at the address / is simple, maybe later we will run more than one application, in which case the placement of all the addresses of our application in / microblog will play the role of the namespace.
Inside the definition of FastCGI, we set the location of the socket file in the / tmp folder and the path to our runp-sqlite.fcgi file, which Lighttpd will launch to start FastCGI processes. The check-local option tells Lighttpd to send FastCGI requests to the server, even if the requested path matches a file on disk. max-procs limits the number of FastCGI processes to 1, which is quite enough for a small server, and also helps to avoid potential problems with several simultaneous write requests to the sqlite database.
The alias.url section reflects static queries for the local path to the corresponding file ..
The runp-sqlite.fcgi script is very similar to the one we used on CentOS, only without overwriting the database settings:
from flup.server.fcgi import WSGIServer from app import app if __name__ == '__main__': WSGIServer(app).run() Some required fixes
Testing this setup will indicate several problems with our application.
Firstly, our javascript and css files are not loaded. The problem is easily solved by looking at the source of the page on the login page. In the base.html template, these files are given in absolute paths that begin with / static, however, we added the prefix / microblog to our addresses. For example, here’s a link to a CSS file:
<link href="/static/css/bootstrap.min.css" rel="stylesheet" media="screen"> At that time, when in fact we would like to see /microblog/static/bootstrap.min.css here, and, of course, we cannot simply add / microblog to the base.html template because it will cause our application to fail on development server running on our PC. The solution is to generate these addresses automatically using url_for:
<link href="{{ url_for('.static', filename = 'css/bootstrap.min.css') }}" rel="stylesheet" media="screen"> After the update, all js and css files will be loaded correctly.
But an attempt to log in will reveal a new problem. Immediately after logging in, we get error 500. Fortunately for us, we have logging enabled, so a quick examination of /home/pi/microblog/tmp/microblog.log leads us to the following error:
ProgrammingError: (ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id -1260325776 and this is thread id -1243548560 What is this?We do not run multiple threads, but it is obvious that our application still uses multiple threads. The only difference between the server for development and this is that here we use the flup for the FastCGI server. Looking into the source code of this module, we learn that by default, flup starts a multi-threaded server.
A possible solution would be to use a single FastCGI server thread provided by flup. However, this can hurt performance in the case of multiple concurrent connections. Another more promising way to solve this problem is to allow multithreading in sqlite (which seems to be fully supported ). Multithreading can be enabled in SQLAlchemy by setting the check_same_thread parameter to False in our config.py configuration file:
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + os.path.join(basedir, 'app.db') + '?check_same_thread=False' With this change, we allowed to run many threads to make changes to the database, while the sqlite library assumes synchronization of multiple access.
And finally, we run our app on the Raspberry Pi!
Install application updates
To install updates, we can simply log in to the Raspberry Pi and call the commands that update the source code, database and translation files:
$ cd microblog $ git pull $ ./db_upgrade.py $ ./tr_compile.py If you have configured a passwordless entry to your Pi, then you can easily write a script that executes all of this in one command.
Conclusion
The updated application is available, as always, on github . Or you can download it as a zip archive by the link:
Download microblog 0.17 .
In the next article, we will look at deployment on a cloud service, as an alternative to the traditional approach that we reviewed today.
See you later!
Miguel
Source: https://habr.com/ru/post/237489/
All Articles