Mega-Tutorial Flask, Part 15: Ajax

This is the fifteenth article in the series, where I describe my experience of writing a Python web application using the Flask mic framework.

The purpose of this guide is to develop a fairly functional microblog application, which I decided to call microblog, in the absence of originality.



This will be the last article on the topic of internationalization and localization (I18n and L10n), which we will summarize our efforts to increase the availability of our microblog application for non-English speaking users.
')
In this article, we will leave the “comfort zone” of server development, to which we have become accustomed, and will start working on the functionality for which the server and client components are equally important. Have you ever seen the "Translate" button that some sites show next to user content? These links automatically translate content into the user's native language in real time. The translated content is usually inserted below the original. Google uses this approach for search results in a foreign language, Facebook for posts. Today we will add this behavior to our microblog!

Server-side vs. Client-side

In the traditional model of client-server interaction, which we have followed so far, we have a client (client's web browser) sending requests to the server. The request simply requests a web page, as in the case when you click the “My Profile” link, or it can perform some action on the server, as in the case when the user edits his profile and clicks “Send”. Regardless of the type, the server responds to the request by sending a new page to the client directly or via a redirect. Then the browser replaces the current page with a new one. This cycle is repeated as long as the user remains on the site. We call this model “server” because the server does all the work of generating pages, but the client simply displays the pages as they are received.

In the “client” model, we still have a browser that sends requests to the server. The server responds with a page that is similar to what happened when using the “server” model, but not all data is represented by HTML, code is also present there, mainly in javascript. As soon as the client receives the page, it displays it and then executes the code that came instead of with the page. Now we have an active client that can work without constant interaction with the server. In the ideal case, the application is downloaded to the client upon initial download, and then it runs and runs on the client even without refreshing the page, interacting with the server only to receive and save data. This type of application is called Single Page Applications or SPA.

Most real-world applications are a combination of these two approaches. Our application, at the moment, works entirely on the server, but today we will add some logic on the client side. In order to translate posts in real time, the browser will send requests to the server, but the server will respond with translated text without causing the page to refresh on the client. The client will then dynamically add the translation to the current page. This technique is known as Ajax , short for Asynchronous Javascript and XML (even though today XML is often replaced with JSON).

Translation of custom content

At the moment, our support for foreign languages ​​is quite good, thanks to Flask-Babel. We can publish our application in a huge number of languages, you just need to find translators ready to help us.

But we missed one moment. We made the application available to users who speak different languages, so now, perhaps, we will begin to appear and write in different languages. And now, our users, looking at the records, are likely to encounter records in languages ​​that they do not understand. It would be nice to offer an automatic translation feature, right? The quality of automatic translation remains not very high, but in most cases it is enough to understand the meaning of what has been written, so that all our users (including us) will benefit from having such a function.

This function is ideal for implementing it with Ajax. Suppose that our main page may contain several posts in different languages. If we implemented the translation of posts in the traditional way, a request to translate a post would cause a replacement of the current page with a new page with the translation of the selected post. After reading, the user would have to press the "Back" button to return to the list of all posts. The fact is that requesting translation of a post is not an important enough task to cause a full page refresh. It would be better if the translated text were simply inserted under the original text of the post, leaving the rest of the content intact. Therefore, today we are implementing our first Ajax service!

The implementation of automatic translation will require several steps from us. First we need to define the language of the post that we are going to translate. Knowing the language of the post, we can find out whether its translation is required for a specific user, because we know the language chosen by the user. If a translation is necessary and the user wishes to see it, we will use our Ajax translation service that runs on the server. As a result, client-side javascript will add a translation of the post to the page.

Post Language Definition

Our first task is to determine the source language of the post. It is not always possible to accurately determine the language, so we just try to do everything that is possible for this. To solve this problem, we use the module guess-language . So let's install it. Linux and Mac OS X users:

flask/bin/pip install guess-language 

Windows users:

 flask\Scripts\pip install guess-language 

With this module, we scan the text of each post, trying to determine its language. Since we would not like to scan one and tezhe posts over and over again, we will do it once for each post when sent by the user. Then we will save information about the language of the post along with the post itself in our database.

Let's add a language field to our Posts table:

 class Post(db.Model): __searchable__ = ['body'] id = db.Column(db.Integer, primary_key = True) body = db.Column(db.String(140)) timestamp = db.Column(db.DateTime) user_id = db.Column(db.Integer, db.ForeignKey('user.id')) language = db.Column(db.String(5)) 

With every change in the database, we should not forget about migrations:

 $ ./db_migrate.py New migration saved as microblog/db_repository/versions/005_migration.py Current database version: 5 

Now that we have a place to save information about the post language, let's add the definition of the language to the process of adding a new post:

 from guess_language import guessLanguage @app.route('/', methods = ['GET', 'POST']) @app.route('/index', methods = ['GET', 'POST']) @app.route('/index/<int:page>', methods = ['GET', 'POST']) @login_required def index(page = 1): form = PostForm() if form.validate_on_submit(): language = guessLanguage(form.post.data) if language == 'UNKNOWN' or len(language) > 5: language = '' post = Post(body = form.post.data, timestamp = datetime.utcnow(), author = g.user, language = language) db.session.add(post) db.session.commit() flash(gettext('Your post is now live!')) return redirect(url_for('index')) posts = g.user.followed_posts().paginate(page, POSTS_PER_PAGE, False) return render_template('index.html', title = 'Home', form = form, posts = posts) 

If it is not possible to find out the language of the post or an unexpectedly long result is returned, we save an empty string in the language field. This will be a marker of the fact that the language of the post is unknown to us.

Displaying the "Translate" link

The next step is to display the "Translate" link for each post written in a language other than the language used by the user (file app / templates / post.html):

 {% if post.language != None and post.language != '' and post.language != g.locale %} <div><a href="#">{{ _('Translate') }}</a></div> {% endif %} 

We did this in the post.html template, thereby adding translation functionality to all pages of the displaying posts. The “Translate” link will be displayed if we can define the language of the post and the specific language is different from the language returned by the localeselector function from Flask-Babel and available to us through g.locale.

This link requires the addition of a new text, the words "Translate", which must be included in our translation files, including. we immediately wrap the string in the _ () function to mark this string for Flask-Babel. To translate this word, we need to update our language reference (tr_update.py), translate it using poedit and compile (tr_compile.py), as we did in the previous article .

At the moment we don’t know to the end what we should do to launch the translation, so the link will be a stub for now.

Translation Services

Before moving on, we need to pick up a translation service.

There are many services that provide translation services, but unfortunately, most of them are either paid or have significant limitations.

The two leading translation services are Google Translate and Microsoft Translator . Both are paid, but Microsoft offers a free plan with a small volume of translations. Google offered free translation earlier, but no more. And it simplifies our choice of service.

Using the Microsoft Translator Service

To use Microsoft Translator there is a set of requirements:

• The application developer must register with the Microsoft Translator app on the Azure Marketplace. Here you can choose a tariff plan (free is at the bottom).
• Then the developer must register the application . The Client ID and Client Secret code will be issued to the registered application, which will be used when sending a request to the service.

After registration is complete, the transfer request process is as follows:

• Get access token by passing client id and secret.
• Call the preferred method - Ajax , HTTP or SOAP , passing the access token and text for translation.

It actually sounds harder than it actually is, t.ch. without getting into details, here’s the function that does all the work and translates the text into another language (file app / translate.py):

 import urllib, httplib import json from flask.ext.babel import gettext from config import MS_TRANSLATOR_CLIENT_ID, MS_TRANSLATOR_CLIENT_SECRET def microsoft_translate(text, sourceLang, destLang): if MS_TRANSLATOR_CLIENT_ID == "" or MS_TRANSLATOR_CLIENT_SECRET == "": return gettext('Error: translation service not configured.') try: # get access token params = urllib.urlencode({ 'client_id': MS_TRANSLATOR_CLIENT_ID, 'client_secret': MS_TRANSLATOR_CLIENT_SECRET, 'scope': 'http://api.microsofttranslator.com', 'grant_type': 'client_credentials' }) conn = httplib.HTTPSConnection("datamarket.accesscontrol.windows.net") conn.request("POST", "/v2/OAuth2-13", params) response = json.loads (conn.getresponse().read()) token = response[u'access_token'] # translate conn = httplib.HTTPConnection('api.microsofttranslator.com') params = { 'appId': 'Bearer ' + token, 'from': sourceLang, 'to': destLang, 'text': text.encode("utf-8") } conn.request("GET", '/V2/Ajax.svc/Translate?' + urllib.urlencode(params)) response = json.loads("{\"response\":" + conn.getresponse().read().decode('utf-8-sig') + "}") return response["response"] except: return gettext('Error: Unexpected error.') 

This function imports two new items from our configuration file, the id and secret codes given to us by Microsoft (the config.py file):

 # microsoft translation service MS_TRANSLATOR_CLIENT_ID = '' #    app id MS_TRANSLATOR_CLIENT_SECRET = '' #   secret 

To use the service, you need to register yourself and register an application to receive data for these configuration variables. Even if you just want to test our microblog, you need to register (it's free).

We added some new text - a few error messages. They need to be localized, so we run tr_update.py, poedit, and tr_compile.py again to update our translation files.

Let's translate something!

So how do we use the translation service? In fact, it is simple. Here is an example:

 $ flask/bin/python Python 2.6.8 (unknown, Jun 9 2012, 11:30:32) >>> from app import translate >>> translate.microsoft_translate('Hi, how are you today?', 'en', 'es') u'¿Hola, cómo estás hoy?' 

Ajax on the server

Now we can translate texts from one language to another, t.ch. we are ready to integrate this functionality into our application.

When a user clicks on the “Translate” link of the post, an Ajax request is sent to the server. We will see how to send this request a little later, and now let's focus on the implementation of processing the request on the server side.

Ajax service on the server will be an ordinary presentation function with one difference - instead of an HTML page or redirect, it will return data, usually in XML or JSON format . Since JSON is somewhat closer to Javascript, we will use this format (file app / views.py):

 from flask import jsonify from translate import microsoft_translate @app.route('/translate', methods = ['POST']) @login_required def translate(): return jsonify({ 'text': microsoft_translate( request.form['text'], request.form['sourceLang'], request.form['destLang']) }) 

There is not much new for us. This submission will process a POST request, which should contain the text for translation and the codes of the source language and the language to which the text should be translated. Since this is a POST request, we access this data as if it were submitted from an HTML form using the request.form dictionary. We call one of our translation functions with this data and, after receiving the translation, we convert the answer into JSON using the Flask function jsonify (). The data that the client will see as an answer to his request will have the following format:

 { "text": "<   >" } 

Ajax on the client

Now we have to call the Ajax view function from our browser, so we return to the post.html template to finish what we started earlier.

We begin by enclosing the text of the post in a span element with a unique id, so that later it is easy to find it in the DOM (file app / templates / post.html):

 <p><strong><span id="post{{post.id}}">{{post.body}}</span></strong></p> 

Notice how we created a unique id using the post id. If the post has id = 3, then the id of the element will be post3.

We will also place the “Translate” link on the span with a unique id in order to be able to hide this link after the translation is displayed:

 <div><span id="translation{{post.id}}"><a href="#">{{ _('Translate') }}</a></span></div> 

Similar to the example above, the translation link will be in the element with id equal to translation3.
And to increase the attractiveness and convenience of the user, we will add an animation informing the user that the translation process on the server is running. It will be hidden after the page loads and will be displayed only during translation, and will also have a unique id:

 <img id="loading{{post.id}}" style="display: none" src="/static/img/loading.gif"> 

So now we have:

• an element with id equal to post <id> containing the text to translate,
• an element with id equal to translation <id>, which contains the “Translate” link and which will later be replaced with the translated text
• and an image with id equal to loading <id>, which will be displayed while the translation is being processed.

The <id> suffix makes these elements unique. We can have as many posts on a page as desired and they will all have their own set of these three values.

Now, by clicking on the “Translate” link, we need to send an Ajax request. To do this, we will create a Javascript function that will do all the work. Let's start by adding a function call by clicking on the "Translate" link:

 <a href="javascript:translate('{{post.language}}', '{{g.locale}}', '#post{{post.id}}', '#translation{{post.id}}', '#loading{{post.id}}');">{{ _('Translate') }}</a> 

The template variables confuse the code a bit, but the function call itself is quite simple. Consider a post with id = 23, written in Spanish and viewed by a user using the English version of the site. In this case, the function will be called like this:

 translate('es', 'en', '#post23', '#translation23', '#loading23') 

Those. the function will receive the languages ​​of the original and the destination, as well as selectors of the three post-related elements.

We added the function call directly to the post.html template, since it is used to display each post. We will create the same function in our base template in order to have access to it on all pages of the application (file app / templates / base.html):

 <script> function translate(sourceLang, destLang, sourceId, destId, loadingId) { $(destId).hide(); $(loadingId).show(); $.post('/translate', { text: $(sourceId).text(), sourceLang: sourceLang, destLang: destLang }).done(function(translated) { $(destId).text(translated['text']) $(loadingId).hide(); $(destId).show(); }).fail(function() { $(destId).text("{{ _('Error: Could not contact server.') }}"); $(loadingId).hide(); $(destId).show(); }); } </script> 

To implement the required functionality, we use jQuery. Let me remind you that we connected jQuery earlier, when we connected Bootstrap.

We start by hiding the “Translate” link and displaying the download indicator.

Then we use the $ .post () function to send an Ajax request to our server. The $ .post function sends a POST request, which is indistinguishable to the server from the request sent by the browser when the form is submitted. The difference is that on the client this request is sent in the background, without the need to reload the entire page. When a response is received from the server, the function will be executed which is an argument to the done () function that will try to insert the received data into the page. This function receives the answer as an argument, t.ch. we simply replace the “Translate” link in the DOM with the translated text, then hide the download indicator, and finally, display the translated text to the user. Done!

If something happened that prevented the client from receiving a response from the server, then the function that is the function argument fail () is executed. In this case, we simply display an error message, which should also be translated into all supported languages ​​and update our translation base.

Unit testing

Remember our testing framework? Always after adding a new code to the application, it is worth assessing whether it makes sense to write tests for this code. Appeal to the translation service can be broken by us in the process of subsequent work on the code, or in consequence of updates in the work of the service itself. Let's write a quick test to make sure that we have access to the service and can get a translation:

 from app.translate import microsoft_translate class TestCase(unittest.TestCase): #... def test_translation(self): assert microsoft_translate(u'English', 'en', 'es') == u'Inglés' assert microsoft_translate(u'Español', 'es', 'en') == u'Spanish' 

At the moment we do not have a framework for testing client side code, so we will not test the full Ajax request cycle.
Running tests should pass without errors:

 $ ./tests.py ..... ---------------------------------------------------------------------- Ran 5 tests in 5.932s OK 

But if you run tests before specifying data access to the Microsoft Translator, you will receive the following:

 $ ./tests.py ....F ====================================================================== FAIL: test_translation (__main__.TestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "./tests.py", line 120, in test_translation assert microsoft_translate(u'English', 'en', 'es') == u'Inglés' AssertionError ---------------------------------------------------------------------- Ran 5 tests in 3.877s FAILED (failures=1) 

Conclusion

On this we finish this article. I hope you read the article no less enjoyable than writing it to me!

Recently I was informed about some problems with the database, when using Flask-WhooshAlchemy for full-text search. In the next article, I use this problem as a pretext for an excursion into some of the debugging techniques I use when working with applications on Flask. Expect part XVI!

Here is a link to the latest microblog version:

Download microblog-0.15.zip.

Or, if you like it better, you can find the source code on GitHub .

Miguel