Flask Mega-Tutorial, Part 18: Deploying on Heroku Cloud

This is the eighteenth article in the series, where I describe my experience of writing a Python web application using the Flask mic framework.

The purpose of this guide is to develop a fairly functional microblogging application, which I decided to call microblog due to the complete lack of originality.



In the previous article, we looked at the traditional hosting option. We saw two topical examples of hosting on Linux servers, first on a regular server running CentOS, and then on the Raspberry Pi minicomputer. Those readers who did not administer Linux systems earlier may have decided that this requires too much effort and could be implemented somehow easier.
')
Today, we’ll see if deploying to the cloud is a solution to the complexity of the process.

But what does it mean to "deploy to the cloud"?

Cloud hosting providers offer a platform on which our application can run. All that is required of the developer is to provide the application, and everything else, including server hardware, the operating system, the language interpreter, and the database, assumes the service.

Sounds too good to be true, right?

We will look at deploying the application on the Heroku platform, one of the most popular cloud hosting platforms. I chose Heroku not only because of its popularity, but also because it provides a free level of service, so we will deploy our application without spending a single cent. If you want to know more about this type of service, and what other providers offer, you can familiarize yourself with the PaaS Wikipedia page.

Hosting on Heroku

Heroku was one of the first platforms to provide PaaS services. At the beginning, she offered hosting services only to Ruby applications, but later support was provided for many other languages ​​such as Java, Node.js and our favorite, Python.

In fact, to deploy an application to Heroku, you only need to download the application using git (you will see how it works very soon). Heroku searches for the Procfile file in the root folder of the application for instructions on how the application should be executed. For Python projects, Heroku also expects to see the requirements.txt file containing a list of required third-party packages.

After downloading the application, we can assume that it is done. Heroku will apply its magic and the application will be available online in seconds. The amount of the invoice at the end of the period directly depends on the computing power consumed by your application, therefore the more users your application has, the more you will have to pay.

Ready to try out Heroku? Let's start!

Create Heroku account

Before placing the application on Heroku, you need to register there. Therefore, click on the link and create an account.

After logging in, you will be taken to the control panel, from where you will be able to manage all your applications. We will not use the control panel extensively, but it does provide a good overview of your account.

Installing Heroku Client

Despite the fact that some tasks can be performed directly from the web interface, there are tasks that can be solved only from the terminal, so we will do everything in the console.

Heroku offers the Heroku Client utility, which we will use to create and manage our application. This utility can be run under Windows, Mac OS X and Linux. If the Heroku toolkit is available for your platform, then this is the easiest way to install the Heroku client.

The first thing we will do with the client is to log in to our account:

$ heroku login 

Heroku will ask you for an email and password for your account. At the first authorization, the client will send your ssh key to the Heroku server.

Subsequent commands can be executed without authorization.

Git Setup

git is the basis for deploying applications on Heroku, so it should also be installed. If you installed the Heroku toolkit, then git is already installed.

To deploy an application to Heroku, it must be present in the local repository, so run the following commands in the console:

 $ git clone git://github.com/miguelgrinberg/microblog.git $ cd microblog 

Creating Heroku App

To create a new Heroku application, just call the create command from the root folder of the application:

 $ heroku create flask-microblog Creating flask-microblog... done, stack is cedar http://flask-microblog.herokuapp.com/ | git@heroku.com:flask-microblog.git 

In addition to setting URLs, this command adds a remote repository ( git remote ) to our repository, which we will soon use to upload application code to the cloud.

Naturally, the name flask-microblog is now taken by me, so think of some other name for your application.

Exclude local file storage

Some functions of our application save information in the form of files on a disk.

And here we are faced with a difficult task. Applications running on Heroku are not available to permanently store files on the disk, because Heroku uses a virtualization platform that does not remember the data as files, the file system is cleared of all files, except the application files, each time the instance is started. Strictly speaking, the application can store temporary files on disk, but should be able to recover these files if they disappear. In addition, if two instances are running, each of them uses its own virtual file system and there is no possibility to share files between them.

This is really bad news for us. For starters, this means that we will not be able to use sqlite as a database.

Our full-text search database Whoosh also stops working, because it stores its data in the form of files.

The third problem point is our logging system. We saved our log in the / tmp folder and now, when working on Heroku, this will also stop working.

So, we identified 3 main problems for which we need to look for solutions.

We will solve the first problem by migrating to the database offered by Heroku, which is based on PostgreSQL .

For the functioning of full-text search, we do not have a ready-made alternative available. We will have to implement full-text search using PostgreSQL functionality, but this will require changes to our application. Certainly a pity, but the solution to this problem would now lead us far away from the topic of the article, therefore, for placement on Heroku, we will simply disable full-text search.

And finally, since we cannot write our logs, we will add our logs to the logging system used by Heroku, which, by the way, is very easy to use, because it sends to the log everything that is output to stdout.

Creating Heroku Database

To create the database, we use the Heroku client:

 $ heroku addons:add heroku-postgresql:dev Adding heroku-postgresql:dev on flask-microblog... done, v3 (free) Attached as HEROKU_POSTGRESQL_ORANGE_URL Database has been created and is available ! This database is empty. If upgrading, you can transfer ! data from another database with pgbackups:restore. Use `heroku addons:docs heroku-postgresql:dev` to view documentation. 

Note that we use the development database, as this is the only free option. For a combat server, you will need to select another database option.

And how does our application know the database connection settings? Heroku puts the database URI in the $ DATABASE_URL environment variable. If you remember, we made changes to our configuration file in the last article, including the value of this variable will be used to connect to the database, as required.

Prohibition of full-text search

To disable full-text search, our application must be able to determine whether it is running on Heroku or not. To do this, we will create a custom environment variable, again using the Heroku client:

 heroku config:set HEROKU=1 

Now the HEROKU environment variable will be set to 1 when our application is running on the Heroku virtual platform.

Now disable full-text search is quite simple. First, let's add a variable to the configuration file (config.py file):

 # Whoosh does not work on Heroku WHOOSH_ENABLED = os.environ.get('HEROKU') is None 

Then, cancel the creation of the full-text search database (file app / models.py):

 from config import WHOOSH_ENABLED if WHOOSH_ENABLED: import flask.ext.whooshalchemy as whooshalchemy whooshalchemy.whoosh_index(app, Post) 

Also add information about the full-text search in g in our before_request handler so that our templates can see it (file app / views.py):

 from config import WHOOSH_ENABLED @app.before_request def before_request(): g.user = current_user if g.user.is_authenticated(): g.user.last_seen = datetime.utcnow() db.session.add(g.user) db.session.commit() g.search_form = SearchForm() g.locale = get_locale() g.search_enabled = WHOOSH_ENABLED 

And finally, we remove the search field in the base template (file app / templates / base.html):

  {% if g.user.is_authenticated() and g.search_enabled %} <form class="navbar-search pull-right" action="{{url_for('search')}}" method="post" name="search">{{g.search_form.hidden_tag()}}{{g.search_form.search(size=20,placeholder=_('Search'),class="search-query")}}</form> {% endif %} 

We correct logging

Under the control of Heroku, everything that is output to the stdout stream immediately gets into the Heroku application log. But logs that are written to files on disk will be inaccessible. So on this platform we need to disable logging to files and use instead a logger who writes errors directly to stdout (file app / __ init__.py):

 if not app.debug and os.environ.get('HEROKU') is None: import logging from logging.handlers import RotatingFileHandler file_handler = RotatingFileHandler('tmp/microblog.log', 'a', 1 * 1024 * 1024, 10) file_handler.setLevel(logging.INFO) file_handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]')) app.logger.addHandler(file_handler) app.logger.setLevel(logging.INFO) app.logger.info('microblog startup') if os.environ.get('HEROKU') is not None: import logging stream_handler = logging.StreamHandler() app.logger.addHandler(stream_handler) app.logger.setLevel(logging.INFO) app.logger.info('microblog startup') 

Web server

Heroku does not provide its web server. Instead, it expects the application to launch its own server on the port, the number of which will be received from the $ PORT environment variable.

We know that the Flask development server is not suitable for production, since it is single-process and single-threaded, so we need a better solution. The Heroku tutorial for Python applications recommends gunicorn , which we will apply.

In our local environment, gunicorn is installed as a regular python module:

 $ flask/bin/pip install gunicorn 

To start it, we need to pass one argument, the name of the Python module that defines the application and the application object itself, separated by a colon.

Let's create a separate Python module for Heroku (file runp-heroku.py):

 #!flask/bin/python from app import app 

Now, for example, if we want to start the gunicorn server locally, using this module, we need to execute the following command:

 $ flask/bin/gunicorn runp-heroku:app 2013-04-24 08:42:34 [31296] [INFO] Starting gunicorn 0.17.2 2013-04-24 08:42:34 [31296] [INFO] Listening at: http://127.0.0.1:8000 (31296) 2013-04-24 08:42:34 [31296] [INFO] Using worker: sync 2013-04-24 08:42:34 [31301] [INFO] Booting worker with pid: 31301 

File requirements.txt

Very soon we will upload our application to Heroku, but first we need to tell the server which modules our application needs to run. On our local PC, we managed dependencies using a virtual environment, installing modules into it using pip.

Heroku does the same. If the requirements.txt file is found in the root folder of the application, then Heroku installs all the modules listed in it using pip.

To create the requirements.txt file, we must use the freeze option when calling pip:

 $ flask/bin/pip freeze > requirements.txt 

The gunicorn server must be added to the list, as well as the psycopg2 driver, which SQLAlchemy requires to connect to the PostgreSQL database. The final form of the requirements.txt file will be as follows:

 Babel==0.9.6 Flask==0.9 Flask-Babel==0.8 Flask-Login==0.1.3 Flask-Mail==0.8.2 Flask-OpenID==1.1.1 Flask-SQLAlchemy==0.16 Flask-WTF==0.8.3 git+git://github.com/miguelgrinberg/Flask-WhooshAlchemy Jinja2==2.6 MySQL-python==1.2.4 psycopg2==2.5 SQLAlchemy==0.7.9 Tempita==0.5.1 WTForms==1.0.3 Werkzeug==0.8.3 Whoosh==2.4.1 blinker==1.2 coverage==3.6 decorator==3.4.0 flup==1.0.3.dev-20110405 guess-language==0.2 gunicorn==0.17.2 python-openid==2.2.5 pytz==2013b speaklater==1.3 sqlalchemy-migrate==0.7.2 

Some of these packages will be unclaimed with the version of our application designed for Heroku, but there is nothing terrible in the presence of unused packages in the system. And it seems to me that it is still better to have a complete list of required packages.

Procfile

The last requirement is to tell Heroku how to start the application. This Heroku requires a Procfile file in the root folder of the application.

This file is quite simple; it simply defines the names of the processes and the commands associated with them (the Procfile file):

 web: gunicorn runp-heroku:app init: python db_create.py && pybabel compile -d app/translations upgrade: python db_upgrade.py && pybabel compile -d app/translations 

The web tag is associated with the web server. Heroku requires this task to run our application.

The other two tasks, called init and upgrade, are user tasks that we will use to work with our application. The init task initializes our application by creating a database and compiling language files. The upgrade task is similar to init, but instead of creating a database, it updates the database to the latest migration.

Deploying the application

Now we will proceed to the most interesting part in which we will place the application in our Heroku account. It's pretty simple, we just use git to send the application:

 $ git push heroku master Counting objects: 307, done. Delta compression using up to 4 threads. Compressing objects: 100% (168/168), done. Writing objects: 100% (307/307), 165.57 KiB, done. Total 307 (delta 142), reused 272 (delta 122) -----> Python app detected -----> No runtime.txt provided; assuming python-2.7.4. -----> Preparing Python runtime (python-2.7.4) -----> Installing Distribute (0.6.36) -----> Installing Pip (1.3.1) -----> Installing dependencies using Pip (1.3.1) ... -----> Discovering process types Procfile declares types -> init, upgrade, web -----> Compiled slug size: 29.6MB -----> Launching... done, v6 http://flask-microblog.herokuapp.com deployed to Heroku To git@heroku.com:flask-microblog.git * [new branch] master -> master 

The heroku tag we use in our git push command was automatically registered in our git repository when we created our application using heroku create. To see how this remote repository is configured, you can run git remote -v in the application folder.

When we initially loaded the application on Heroku, we need to initialize the database and translation files, and for this we need to complete the task init, which we included in our Procfile:

 $ heroku run init Running `init` attached to terminal... up, run.7671 /app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname> module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects compiling catalog 'app/translations/es/LC_MESSAGES/messages.po' to 'app/translations/es/LC_MESSAGES/messages.mo' 

The warning belongs to SQLAlchemy, because it does not like URIs that begin with postgres: // instead of postgresql: //. This URI forms Heroku through the value of the $ DATABASE_URL environment variable, so it is not in our power to change this. It remains to hope that this URI format will work for a long time.

Believe it or not, our application is already available online. In my case, the application is available at flask-microblog.herokuapp.com . You can easily become my follower from my profile page. I do not know exactly how long the application will be available at this address, but nothing prevents you from checking whether it is available or not!

App update

Sooner or later, the time will come to update our application. This will be similar to the initial deployment. First of all, the application will be uploaded to the server using git:

 $ git push heroku master 

Then, the update script is executed:

 $ heroku run upgrade 

Logging

If something abnormal happens to the application, it may be useful to examine the logs. Remember that for Heroku version of the application, we write all the logs in stdout, and Heroku collects them into its own log.

To view the logs, use the Heroku client:

 $ heroku logs 

The above command will output all logs, including the Heroku logs. To view the logs of your application only, run the command:

 $ heroku logs --source app 

Things like call stack and other application errors will all be in this log.

Is it worth it?

Now we have an idea of ​​application deployment to a cloud platform, and therefore we can compare this type of hosting with a traditional hosting option.

In the matter of simplicity, victory over the clouds. At least for Heroku, the application deployment process was very simple. When deploying to a dedicated server or VPS, a lot of preparatory work was needed. Heroku takes care of it and allows us to focus on our application.

The cost issue is a controversial point. The cost of cloud hosting services is usually more expensive than dedicated servers, because you pay not only for the server, but also for the administration services. A typical Heroku tariff plan, which includes two instances and the cheapest production database, will cost $ 85 (this is at the time of this writing. About a year ago - approx. Lane ). On the other hand, if you do a good search, you can easily find a decent VPS for about $ 40 per year.

In the end, it seems to me, the question of choice will be reduced to the choice of what is more important for you: time or money.

The end?

The updated application is available, as always, on github . Or you can download it as a zip archive by the link:

Download microblog 0.18 .

With the deployment of our application in all possible ways, it seems that our excursion comes to an end.

I hope these articles were a useful introduction to the development of a real web application, and that the knowledge base that I threw on you for these 18 articles motivates you to create your own project.

However, I do not put an end, and do not deny the likelihood of articles on the microblog. If, and when, an interesting topic comes to mind, I will write more, but I expect that the frequency of updates will now somewhat subside. From time to time, I can make some minor fixes in the application that do not deserve a separate article in the blog, so you can track these changes on GitHub .

In my blog, I will continue to write articles related to web development and software in general, t.ch. I invite you to follow me on Twitter or Facebook if you have not already done so, and in this way you will be notified of my future articles.

Thank you, once again, for being a loyal reader.

Miguel