Tornado WebSocket Chat for your Django project

Recently, I launched the site backgrounddating.com and wrote about it here on Habrahabr. Of course, I already spoke about some of the technical details of the implementation of this project, but I would like to write about one of the site’s capabilities separately, especially since the documentation (both in Russian and in English) on this topic on the Internet is still quite small. . So, it’s about real-time chat between two users. The task is to allow any user to send messages to other users, and if the recipient of the message has a chat with these users, he immediately saw the incoming messages (otherwise, he could read the messages later: that is, when the chat opens, history of recent posts).

If you need to allow users to communicate not only together, but in groups of any number of people, then this can be done almost elementarily: the described implementation, in fact, is designed for such an extension of functionality.

Immediately clarify that this is not the only way to implement this. You can use another asynchronous web server (for example, node.js), you can use another message queue (or not to use it at all if you like the features of this option: the same web server worker must communicate with users of the same channel). I do not even claim that this option is the best (but in this case it came up better than anyone). In the end, here we will not consider crutches at all (long polling, Flash) for old browsers (and these are almost all versions of IE, for example) that do not support web sockets, and we will not even consider connecting from those browsers that already support the WebSocket protocol, but not the standardized version ( RFC 6455 ), but one of the obsolete ones. For information on how to enable support for the outdated version of “draft 76” (aka “hixie-76”), see the Tornado documentation .

However, what can be said for sure - this method works well, and not in one project, but in many (the described implementation method has been used for a long time, even though there is not very much information about it yet). For example, the server running Background Dating is currently the youngest VPS from Linode (512 MiB of memory), but the load on the processor did not rise by more than 20-40 percent, and the use of RAM is about 30%. And the resources are mainly used by gunicorn (the web server on which Django runs) and PostgreSQL. But there is absolutely nothing surprising, since it is not a secret for anyone that Tornado is easy enough not only to work quickly, but even to cope with the C10k (which was already in the Habrahabr).
')
So, we will use Django 1.4.2, Tornado 2.4, Redis 2.6.5 and PostgreSQL 9.2.1 (in fact, you can use another relational DBMS - Django has many different backends). The standard Redis-py client for Python will be used to connect to Redis, as well as the brükva (asynchronous Redis client for use with Tornado). In order to deploy all this on the server, we will use haproxy and nginx , we will use the gunicorn web server to run the Django project in production, and Supervisor will manage the launch of the web servers for Django and Tornado.

If you feel that you lack theoretical knowledge (for example, you do not quite understand what an asynchronous non-blocking web server is, a message queue, and so on), I recommend first reading the relevant documentation, and if you have questions, write to comments (or me by mail ). I also recommend to pay attention to several discussions on the topic ( 1 , 2 , 3 , 4 , 5 , 6 , 7 ) on the excellent forum of Python-guru Ivan Sagalayev, as well as on the slides of the report “We connect the synchronous framework with asynchronous (for example django)” , which is another Python-guru, Mikhail Korobov, read at DevConf 2011 .

The source code of the considered solution is present both in this article and on GitHub .

Django setup

Create a Django project and go to the directory that appears:

 django-admin.py startproject myproject
 cd myproject /

Now edit the file myproject / settings.py.

First of all, it would be a very good idea to immediately switch to using the Redis backend to store information about user sessions. This should be done in all projects (except for those where for some reason you cannot use Redis, as well as those where many users are simply not planned). To do this, install django-redis-sessions (pip install django-redis-sessions) and just add in the settings:

SESSION_ENGINE = 'redis_sessions.session'

Congratulations, you have just done so that the number of queries to the database when sending requests to the site is reduced by 1 (Redis responds to requests almost instantly). :)

By the way, these settings are best saved in a separate file that you add to gitignore - usually local_settings.py.

There you can transfer the project secret key, for example (SECRET_KEY), as well as database settings. The point here is that firstly, you can save server-specific settings in local_settings.py (database, caching backend, session backend, debug mode settings), and secondly Git (or another version control system) will not store information about keys and passwords.

Accordingly, in order to add such settings to local_settings.py, you simply need to add the following at the end of settings.py:

 try: from local_settings import * except ImportError: pass

Next, do not forget to set up the database (DATABASES), specify the template directory, static files directory and API key (Tornado will receive a request to Django asynchronously and Django will save this message in the database), as well as who need to send requests.

To ensure that when a site is deployed on another server (or when a project is placed elsewhere in the file system), the path to the directories of templates and static files does not need to be changed, it is best to determine the location of the project at launch by adding the following in the settings:

 import os PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))

And then, respectively, to receive other paths based on PROJECT_ROOT.

Static files:

 STATICFILES_DIRS = ( os.path.join(PROJECT_ROOT, "static"), )

Templates:

 TEMPLATE_DIRS = ( os.path.join(PROJECT_ROOT, "templates"), )

API key and address:

 API_KEY = '$0m3-U/\/1qu3-K3Y' SEND_MESSAGE_API_URL = 'http://127.0.0.1:8000/messages/send_message_api'

In order to generate a key, you can use such a simple line in the console (by the way, the emoticon is looking at you from there):

 </ dev / urandom tr -dc _A-Zaz-0-9 |  head -c $ {1: -32}; echo;

It now remains to create the static and templates directories in the myproject directory (where settings.py and local_settings.py are located) and start writing a chat application.

Chat (Django)

Add a new application:

 python manage.py startapp privatemessages

And write the model (privatemessages / models.py):

 from django.db import models from django.db.models.signals import post_save from django.contrib.auth.models import User # Create your models here. class Thread(models.Model): participants = models.ManyToManyField(User) last_message = models.DateTimeField(null=True, blank=True, db_index=True) class Message(models.Model): text = models.TextField() sender = models.ForeignKey(User) thread = models.ForeignKey(Thread) datetime = models.DateTimeField(auto_now_add=True, db_index=True) def update_last_message_datetime(sender, instance, created, **kwargs): """ Update Thread's last_message field when a new message is sent. """ if not created: return Thread.objects.filter(id=instance.thread.id).update( last_message=instance.datetime ) post_save.connect(update_last_message_datetime, sender=Message)

There is nothing complicated here - there are messages and threads (threads). For every two users will create their own thread, and all their messages will relate to this thread. When a new message is created, the date and time of the last message for the corresponding thread is updated.

Now it's time to add the application to INSTALLED_APPS in settings.py and synchronize the models with the database:

 python manage.py syncdb

Let's open privatemessages / views.py and write 4 views:

send_message_view - to send a message via Django (for example, if this is the first message between two people)
send_message_api_view - to send messages through Tornado
messages_view - to view the list of interlocutors (sorted by descending date and time of the last message)
chat_view - for chat (Django will simply return the page with those messages that already exist in the database, and then the browser will connect to the Tornado server via WS and receive / send new messages in real time, without reloading the page and without additional HTTP- requests)

 # Create your views here. import json import redis from django.shortcuts import render_to_response, get_object_or_404 from django.http import HttpResponse, HttpResponseRedirect from django.template import RequestContext from django.core.urlresolvers import reverse from django.utils import timezone from django.views.decorators.csrf import csrf_exempt from django.conf import settings from django.contrib.auth.models import User from privatemessages.models import Thread, Message from privatemessages.utils import json_response, send_message def send_message_view(request): if not request.method == "POST": return HttpResponse("Please use POST.") if not request.user.is_authenticated(): return HttpResponse("Please sign in.") message_text = request.POST.get("message") if not message_text: return HttpResponse("No message found.") if len(message_text) > 10000: return HttpResponse("The message is too long.") recipient_name = request.POST.get("recipient_name") try: recipient = User.objects.get(username=recipient_name) except User.DoesNotExist: return HttpResponse("No such user.") if recipient == request.user: return HttpResponse("You cannot send messages to yourself.") thread_queryset = Thread.objects.filter( participants=recipient ).filter( participants=request.user ) if thread_queryset.exists(): thread = thread_queryset[0] else: thread = Thread.objects.create() thread.participants.add(request.user, recipient) send_message( thread.id, request.user.id, message_text, request.user.username ) return HttpResponseRedirect( reverse('privatemessages.views.messages_view') ) @csrf_exempt def send_message_api_view(request, thread_id): if not request.method == "POST": return json_response({"error": "Please use POST."}) api_key = request.POST.get("api_key") if api_key != settings.API_KEY: return json_response({"error": "Please pass a correct API key."}) try: thread = Thread.objects.get(id=thread_id) except Thread.DoesNotExist: return json_response({"error": "No such thread."}) try: sender = User.objects.get(id=request.POST.get("sender_id")) except User.DoesNotExist: return json_response({"error": "No such user."}) message_text = request.POST.get("message") if not message_text: return json_response({"error": "No message found."}) if len(message_text) > 10000: return json_response({"error": "The message is too long."}) send_message( thread.id, sender.id, message_text ) return json_response({"status": "ok"}) def messages_view(request): if not request.user.is_authenticated(): return HttpResponse("Please sign in.") threads = Thread.objects.filter( participants=request.user ).order_by("-last_message") if not threads: return render_to_response('private_messages.html', {}, context_instance=RequestContext(request)) r = redis.StrictRedis() user_id = str(request.user.id) for thread in threads: thread.partner = thread.participants.exclude(id=request.user.id)[0] thread.total_messages = r.hget( "".join(["thread_", str(thread.id), "_messages"]), "total_messages" ) return render_to_response('private_messages.html', { "threads": threads, }, context_instance=RequestContext(request)) def chat_view(request, thread_id): if not request.user.is_authenticated(): return HttpResponse("Please sign in.") thread = get_object_or_404( Thread, id=thread_id, participants__id=request.user.id ) messages = thread.message_set.order_by("-datetime")[:100] user_id = str(request.user.id) r = redis.StrictRedis() messages_total = r.hget( "".join(["thread_", thread_id, "_messages"]), "total_messages" ) messages_sent = r.hget( "".join(["thread_", thread_id, "_messages"]), "".join(["from_", user_id]) ) if messages_total: messages_total = int(messages_total) else: messages_total = 0 if messages_sent: messages_sent = int(messages_sent) else: messages_sent = 0 messages_received = messages_total-messages_sent partner = thread.participants.exclude(id=request.user.id)[0] tz = request.COOKIES.get("timezone") if tz: timezone.activate(tz) return render_to_response('chat.html', { "thread_id": thread_id, "thread_messages": messages, "messages_total": messages_total, "messages_sent": messages_sent, "messages_received": messages_received, "partner": partner, }, context_instance=RequestContext(request))

And here is privatemessages / utils.py:

 import json import redis from django.utils import dateformat from privatemessages.models import Message def json_response(obj): """ This function takes a Python object (a dictionary or a list) as an argument and returns an HttpResponse object containing the data from the object exported into the JSON format. """ return HttpResponse(json.dumps(obj), content_type="application/json") def send_message(thread_id, sender_id, message_text, sender_name=None): """ This function takes Thread object id (first argument), sender id (second argument), message text (third argument) and can also take sender's name. It creates a new Message object and increases the values stored in Redis that represent the total number of messages for the thread and the number of this thread's messages sent from this specific user. If a sender's name is passed, it also publishes the message in the thread's channel in Redis (otherwise it is assumed that the message was already published in the channel). """ message = Message() message.text = message_text message.thread_id = thread_id message.sender_id = sender_id message.save() thread_id = str(thread_id) sender_id = str(sender_id) r = redis.StrictRedis() if sender_name: r.publish("".join(["thread_", thread_id, "_messages"]), json.dumps({ "timestamp": dateformat.format(message.datetime, 'U'), "sender": sender_name, "text": message_text, })) for key in ("total_messages", "".join(["from_", sender_id])): r.hincrby( "".join(["thread_", thread_id, "_messages"]), key, 1 )

You can read about tz.activate () in the Django documentation for working with time zones. Since browsers do not provide information about the time zone (for example, unlike language information), we need to create a cookie on the client’s side with the name timezone — this way Django can display the date and time of each message in the time zone in which moment is the user.

In order to calculate the time zone on the client side, we will use the jstz library (do not forget to place jstz.min.js in the directory myproject / static). There are other solutions to determine the user's time zone: for example, you can take the most likely time zone based on GeoIP or even compare the local time on the user's computer with the local time on the server (since we know that the time is correct on the server) - this solution allows you to find out the user's actual time zone in the event that a completely different time zone is selected on his system, but the time has been manually transferred to correspond to the time zone.

But in this case, let's assume that if the user has a wrong time zone, then most likely the inaccuracy of time suits him (otherwise, it will be an additional reminder to change the settings).

Also note that you will need to install pytz (pip install pytz) - this is needed in order to get the time zone from a string in the Olson format ( IANA time zone database ).

When the send_messages function adds a new message to the database, it also updates the hash with the number of messages for the thread in Redis. In this hash two keys are incremented - one of them represents the total number of messages in the thread, and the other represents the number of messages from this user. These keys are used when displaying the total number of messages, as well as the number of received and sent messages (the number of received is just the total number minus the number of sent).

When a user opens a chat and his browser opens a WS connection to the Tornado server, the Tornado server subscribes to the thread channel in Redis, and when new messages appear, it immediately sends them to the user.

If you are not familiar with the Redis Pub / Sub implementation (SUBSCRIBE, UNSUBSCRIBE, and PUBLISH), then you can read about it in the Redis documentation .

When the send_messages function calls send_message_view and sends, in particular, the name of the sender, the function publishes a message on the channel of this thread to Redis. In the case of send_message_api_view, the sender's name is not transmitted, and it is assumed that the message has already been sent to the channel of this thread, even before the send_message_api_view call (this is done so that messages are transmitted as quickly as possible between chatting users - saving to the database asynchronously , after the publication on the channel).

Now we will create a privatemessages / urls.py file and set the urlpatterns for the Django methods:

 from django.conf.urls import patterns, url urlpatterns = patterns('privatemessages.views', url(r'^send_message/$', 'send_message_view'), url(r'^send_message_api/(?P<thread_id>\d+)/$', 'send_message_api_view'), url(r'^chat/(?P<thread_id>\d+)/$', 'chat_view'), url(r'^$', 'messages_view'), )

And, of course, include them in root URLconf (myproject / urls.py):

 from django.conf.urls import patterns, include, url # something else urlpatterns = patterns('', # something else url(r'^messages/', include('privatemessages.urls')), # something else )

In the template directory you need to place chat.html and private_messages.html. Also add the base template base.html.

base.html

 <!DOCTYPE html> <html> <head> <link rel="stylesheet" type="text/css" href="/static/privatemessages.css"> <script type="text/javascript" src="http://yandex.st/jquery/1.8.3/jquery.min.js"></script> <script type="text/javascript" src="/static/jstz.min.js"></script> <script type="text/javascript" src="/static/privatemessages.js"></script> {% block head %}{% endblock %} <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <title>{% block title %} {% endblock title %}</title> </head> <body> {% block content %}{% endblock content %} </body> </html>

chat.html

 {% extends "base.html" %} {% block title %}{{ partner.username }}{% endblock %} {% block head %} <script type="text/javascript"> $(document).ready(function() { activate_chat({{ thread_id }}, "{{ user.username }}", { "total": {{ messages_total }}, "sent": {{ messages_sent }}, "received": {{ messages_received }} }); }); </script> {% endblock %} {% block content %} {% load pluralize %} <div class="chat"> <div class="partner"> <p class="name">{{ partner.username }}</p> <p class="messages"><span class="total">{{ messages_total }}</span> {{ messages_total|rupluralize:",," }} (<span class="received">{{ messages_received }}</span> , <span class="sent">{{ messages_sent }}</span> )</p> </div> <div class="conversation"> {% for message in thread_messages reversed %} <div class="message"> {% if message.sender == user %}<p class="author we"><span class="datetime">{{ message.datetime|date:"dmY H:i:s" }}</span> {{ user.username }}:</p>{% else %}<p class="author partner"><span class="datetime">{{ message.datetime|date:"dmY H:i:s" }}</span> {{ partner.username }}:</p>{% endif %} <p class="message">{{ message.text|linebreaksbr }}</p> </div> {% endfor %} </div> <form class="message_form"> <div class="compose"> <textarea rows="1" cols="30" id="message_textarea"></textarea> </div> <div class="send"> <button class="btn" type="button"></button> <p>        Ctrl + Enter.</p> </div> </form> </div> {% endblock content %}

private_messages.html

 {% extends "base.html" %} {% block content %} {% load pluralize %} <div class="private_messages"> <h1></h1> <div class="partners"> {% for thread in threads %} <p><a href="{% url privatemessages.views.chat_view thread.id %}">{{ thread.partner.username }} ({{ thread.total_messages|default_if_none:"0" }} {{ thread.total_messages|rupluralize:",," }})</a></p> {% empty %} <p>   .</p> {% endfor %} </div> <h1> </h1> <form action="{% url privatemessages.views.send_message_view %}" method="post" class="new_message"> {% csrf_token %} <p class="name"><input name="recipient_name" placeholder=" "></p> <p><textarea name="message" placeholder=""></textarea></p> <p><input type="submit" value=""></p> </form> </div> {% endblock content %}

And create a privatemessages / templatetags directory with __init__.py and pluralize.py files.

privatemessages / templatetags / pluralize.py ( filter author - V @ s3K):

 from django import template register = template.Library() @register.filter def rupluralize(value, arg): args = arg.split(",") try: number = abs(int(value)) except TypeError: number = 0 a = number % 10 b = number % 100 if (a == 1) and (b != 11): return args[0] elif (a >= 2) and (a <= 4) and ((b < 10) or (b >= 20)): return args[1] else: return args[2]

Add styles (myproject / static / privatemessages.css):

 html, body { height: 100%; margin: 0; } body { font-family: Geneva, Arial, Helvetica, sans-serif; font-size: 14px; color: #000; background: #fff; } textarea, form p.name input { border: 1px #d4d4d4 solid; } form.new_message p { margin: 4px 0; } form.new_message p.name input, form.new_message textarea { width: 300px; } form.new_message textarea { height: 100px; } textarea:focus, input.name:focus { border-color: #cacaca; -webkit-box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset, 0 0 3px rgba(150, 150, 150, 0.5); -moz-box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset, 0 0 3px rgba(150, 150, 150, 0.5); box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset, 0 0 3px rgba(150, 150, 150, 0.5); } div.private_messages { padding: 10px; } div.private_messages h1 { margin-top: 0; } div.private_messages div.partners { margin-bottom: 30px; } div.chat { height: 100%; min-height: 400px; min-width: 600px; position: relative; background-color: #e0e0e0; } div.chat div.partner { height: 50px; padding: 5px; } div.chat div.partner p { margin: 0; } div.chat div.partner p.name { font-weight: bold; margin-bottom: 3px; } div.chat div.conversation { position: absolute; top: 50px; left: 5px; right: 5px; bottom: 140px; overflow: auto; padding: 0 5px; border-style: solid; border-color: #eee; border-width: 10px 0; background-color: #fff; -webkit-border-radius: 7px; -moz-border-radius: 7px; border-radius: 7px; } div.chat div.conversation div.message { padding: 5px 0; } div.chat div.conversation div.message p { margin: 0; } div.chat div.conversation div.message p.author.partner { color: #002c64; } div.chat div.conversation div.message p.author.we { color: #216300; } div.chat div.conversation div.message p.author span.datetime { font-size: 12px; } div.chat form.message_form { position: absolute; margin: 0; left: 0; right: 0; bottom: 5px; height: 130px; } div.chat form.message_form div.outdated_browser_message { margin: 10px 5px; } div.chat form.message_form div.compose { float: left; height: 100%; width: 80%; padding: 0 5px; -webkit-box-sizing: border-box; -moz-box-sizing: border-box; box-sizing: border-box; } div.chat form.message_form div.compose textarea { width: 100%; height: 100%; margin: 0; -webkit-box-sizing: border-box; -moz-box-sizing: border-box; box-sizing: border-box; resize: none; } div.chat form.message_form div.send { float: left; height: 100%; width: 20%; min-width: 100px; padding-right: 5px; -webkit-box-sizing: border-box; -moz-box-sizing: border-box; box-sizing: border-box; } div.chat form.message_form div.send p { margin-top: 5px; color: #333; } div.chat form.message_form div.send button { width: 100%; -webkit-box-sizing: border-box; -moz-box-sizing: border-box; box-sizing: border-box; }

And JavaScript (myproject / static / privatemessages.js):

 function getCookie(name) { var cookieValue = null; if (document.cookie && document.cookie != '') { var cookies = document.cookie.split(';'); for (var i = 0; i < cookies.length; i++) { var cookie = jQuery.trim(cookies[i]); // Does this cookie string begin with the name we want? if (cookie.substring(0, name.length + 1) == (name + '=')) { cookieValue = decodeURIComponent(cookie.substring(name.length + 1)); break; } } } return cookieValue; } function setCookie(key, value) { document.cookie = escape(key) + '=' + escape(value); } function getNumEnding(iNumber, aEndings) { var sEnding, i; iNumber = iNumber % 100; if (iNumber>=11 && iNumber<=19) { sEnding=aEndings[2]; } else { i = iNumber % 10; switch (i) { case (1): sEnding = aEndings[0]; break; case (2): case (3): case (4): sEnding = aEndings[1]; break; default: sEnding = aEndings[2]; } } return sEnding; } var timezone = getCookie('timezone'); if (timezone == null) { setCookie("timezone", jstz.determine().name()); } function activate_chat(thread_id, user_name, number_of_messages) { $("div.chat form.message_form div.compose textarea").focus(); function scroll_chat_window() { $("div.chat div.conversation").scrollTop($("div.chat div.conversation")[0].scrollHeight); } scroll_chat_window(); var ws; function start_chat_ws() { ws = new WebSocket("ws://127.0.0.1:8888/" + thread_id + "/"); ws.onmessage = function(event) { var message_data = JSON.parse(event.data); var date = new Date(message_data.timestamp*1000); var time = $.map([date.getHours(), date.getMinutes(), date.getSeconds()], function(val, i) { return (val < 10) ? '0' + val : val; }); $("div.chat div.conversation").append('<div class="message"><p class="author ' + ((message_data.sender == user_name) ? 'we' : 'partner') + '"><span class="datetime">' + time[0] + ':' + time[1] + ':' + time[2] + '</span> ' + message_data.sender + ':</p><p class="message">' + message_data.text.replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/\n/g, '<br />') + '</p></div>'); scroll_chat_window(); number_of_messages["total"]++; if (message_data.sender == user_name) { number_of_messages["sent"]++; } else { number_of_messages["received"]++; } $("div.chat p.messages").html('<span class="total">' + number_of_messages["total"] + '</span> ' + getNumEnding(number_of_messages["total"], ["", "", ""]) + ' (<span class="received">' + number_of_messages["received"] + '</span> , <span class="sent">' + number_of_messages["sent"] + '</span> )'); } ws.onclose = function(){ // Try to reconnect in 5 seconds setTimeout(function() {start_chat_ws()}, 5000); }; } if ("WebSocket" in window) { start_chat_ws(); } else { $("form.message_form").html('<div class="outdated_browser_message"><p><em>!</em>    . ,    :</p><ul><li> <em>Android</em>: <a href="http://www.mozilla.org/ru/mobile/">Firefox</a>, <a href="http://www.google.com/intl/en/chrome/browser/mobile/android.html">Google Chrome</a>, <a href="https://play.google.com/store/apps/details?id=com.opera.browser">Opera Mobile</a></li><li> <em>Linux</em>, <em>Mac OS X</em>  <em>Windows</em>: <a href="http://www.mozilla.org/ru/firefox/fx/">Firefox</a>, <a href="https://www.google.com/intl/ru/chrome/browser/">Google Chrome</a>, <a href="http://ru.opera.com/browser/download/">Opera</a></li></ul></div>'); return false; } function send_message() { var textarea = $("textarea#message_textarea"); if (textarea.val() == "") { return false; } if (ws.readyState != WebSocket.OPEN) { return false; } ws.send(textarea.val()); textarea.val(""); } $("form.message_form div.send button").click(send_message); $("textarea#message_textarea").keydown(function (e) { // Ctrl + Enter if (e.ctrlKey && e.keyCode == 13) { send_message(); } }); }

When deploying to the server, do not forget to specify the address at which the Tornado server will be available (or even make it into a separate variable).

Chat (Tornado)

Write a new Tornado application and save it in privatemessages / tornadoapp.py:

 import datetime import json import time import urllib import brukva import tornado.web import tornado.websocket import tornado.ioloop import tornado.httpclient from django.conf import settings from django.utils.importlib import import_module session_engine = import_module(settings.SESSION_ENGINE) from django.contrib.auth.models import User from privatemessages.models import Thread c = brukva.Client() c.connect() class MainHandler(tornado.web.RequestHandler): def get(self): self.set_header('Content-Type', 'text/plain') self.write('Hello. :)') class MessagesHandler(tornado.websocket.WebSocketHandler): def __init__(self, *args, **kwargs): super(MessagesHandler, self).__init__(*args, **kwargs) self.client = brukva.Client() self.client.connect() def open(self, thread_id): session_key = self.get_cookie(settings.SESSION_COOKIE_NAME) session = session_engine.SessionStore(session_key) try: self.user_id = session["_auth_user_id"] self.sender_name = User.objects.get(id=self.user_id).username except (KeyError, User.DoesNotExist): self.close() return if not Thread.objects.filter( id=thread_id, participants__id=self.user_id ).exists(): self.close() return self.channel = "".join(['thread_', thread_id,'_messages']) self.client.subscribe(self.channel) self.thread_id = thread_id self.client.listen(self.show_new_message) def handle_request(self, response): pass def on_message(self, message): if not message: return if len(message) > 10000: return c.publish(self.channel, json.dumps({ "timestamp": int(time.time()), "sender": self.sender_name, "text": message, })) http_client = tornado.httpclient.AsyncHTTPClient() request = tornado.httpclient.HTTPRequest( "".join([ settings.SEND_MESSAGE_API_URL, "/", self.thread_id, "/" ]), method="POST", body=urllib.urlencode({ "message": message.encode("utf-8"), "api_key": settings.API_KEY, "sender_id": self.user_id, }) ) http_client.fetch(request, self.handle_request) def show_new_message(self, result): self.write_message(str(result.body)) def on_close(self): try: self.client.unsubscribe(self.channel) except AttributeError: pass def check(): if self.client.connection.in_progress: tornado.ioloop.IOLoop.instance().add_timeout( datetime.timedelta(0.00001), check ) else: self.client.disconnect() tornado.ioloop.IOLoop.instance().add_timeout( datetime.timedelta(0.00001), check ) application = tornado.web.Application([ (r"/", MainHandler), (r'/(?P<thread_id>\d+)/', MessagesHandler), ])

Almost everything in this application is executed asynchronously, using tornado.ioloop. The exception is the authorization of the user by session identifier (and getting his name from the database). This is a blocking operation, that is, at this moment other users connected to this Tornado server will be waiting. If you need to, it is easy to change, but, firstly, not the fact that it is required, and secondly, think about whether this is the right decision. On this occasion, the guys from Friendfeed (the Tornado developers) spoke out more than once. For example, here’s a quote from Ben Darnell’s post:

Friendfeed uses mysql, with the standard synchronous MySQLdb module (http://bret.appspot.com/entry/how-friendfeed-uses-mysql). If you’re not thinking about what you need, it’s not worth it. managing callbacks. If you want to make it, it’s possible to keep the amount of time (long polling). It is not clear that it makes it possible to use it.

That is, roughly speaking, if your database is slow, it means that the server in any case cannot cope with the load, and asynchronous start will not do much for it. , , — , , .

Tornado- management-. Django ORM .

management- . privatemessages management, commands.

privatemessages/management privatemessages/management/commands __init__.py ( - , , ).

privatemessages/management/commands/starttornadoapp.py:

 import signal import time import tornado.httpserver import tornado.ioloop from django.core.management.base import BaseCommand, CommandError from privatemessages.tornadoapp import application class Command(BaseCommand): args = '[port_number]' help = 'Starts the Tornado application for message handling.' def sig_handler(self, sig, frame): """Catch signal and init callback""" tornado.ioloop.IOLoop.instance().add_callback(self.shutdown) def shutdown(self): """Stop server and add callback to stop i/o loop""" self.http_server.stop() io_loop = tornado.ioloop.IOLoop.instance() io_loop.add_timeout(time.time() + 2, io_loop.stop) def handle(self, *args, **options): if len(args) == 1: try: port = int(args[0]) except ValueError: raise CommandError('Invalid port number specified') else: port = 8888 self.http_server = tornado.httpserver.HTTPServer(application) self.http_server.listen(port, address="127.0.0.1") # Init signals handler signal.signal(signal.SIGTERM, self.sig_handler) # This will also catch KeyboardInterrupt exception signal.signal(signal.SIGINT, self.sig_handler) tornado.ioloop.IOLoop.instance().start()

, (, SIGINT, Ctrl + C) - , . .

( myproject privatemessages, manage.py) development- Django Tornado.

python manage.py runserver
python manage.py starttornadoapp

, screen (, , SSH SSH-).

8000 Django, 8888 Tornado.

:

http://127.0.0.1:8000/messages/

( ).

, Django-. , , . — enable the administrative interface and log in through it.

Deployment

In order to run your application on the server, we can use the following software:

haproxy - load balancer, will work on port 80
nginx - distribution of static files + reverse proxy for gunicorn, will work on port 8100
Supervisor - starting the Tornado server on ports 8000–8003 (4 processes) and the Django server (gunicorn can be used) on port 8150

Supervisor. ( Supervisor Ubuntu) /etc/supervisor/conf.d django.conf tornadoapp.conf. , Supervisor ( include files = /etc/supervisor/conf.d/*.conf).

django.conf

[program:django]
command=gunicorn_django --workers 4 -b 127.0.0.1:8150
directory=/home/yourusername/myproject
user=yourusername
autostart=true
autorestart=true

tornadoapp.conf

process_name = tornado-%(process_num)s
user = yourusername
directory = /home/yourusername/myproject
command = python manage.py starttornadoapp %(process_num)s
# Increase numprocs to run multiple processes on different ports.
# Note that the chat demo won't actually work in that configuration
# because it assumes all listeners are in one process.
numprocs = 4
numprocs_start = 8000
autostart = true
autorestart = true

Now run supervisor and see the supervisorctl status command output.

Make sure that all 5 processes (gunicorn will be displayed as one process) show the RUNNING state and the uptime is the same as how much time has passed since Supervisor was started.

I also recommend making sure that both Django servers and Tornado servers work. To do this, we can open the SSH tunnel to the server and simply try to open your site in the browser, while accessing the selected port on the local address.

That is, for example:

ssh -L 8000: localhost: 8150 someverycoolserver.com

127.0.0.1:8000 (SSH- 127.0.0.1:8150 — -, ).

Tornado-. 8000—8003.

Django Tornado , nginx haproxy.

nginx:

server {
    listen 127.0.0.1:8100;
    server_name someverycoolserver.com;

    # no security problem here, since / is always passed to upstream
    root /home/yourusername/myproject/myproject/static/;

    ## Compression
    # src: http://www.ruby-forum.com/topic/141251
    # src: http://wiki.brightbox.co.uk/docs:nginx

    gzip on;
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_min_length 1100;
    gzip_buffers 16 8k;
    gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    
    # Some version of IE 6 don't handle compression well on some mime-types, so just disable for them
    gzip_disable "MSIE [1-6].(?!.*SV1)";
    
    # Set a vary header so downstream proxies don't send cached gzipped content to IE6
    gzip_vary on;
    ## /Compression

    location /static/admin/ {
        # this changes depending on your python version
        root /usr/local/lib/python2.7/dist-packages/django/contrib/admin/;
     }

    location /robots.txt {
        alias /home/yourusername/myproject/myproject/robots.txt;
     }

    location /favicon.ico {
        alias /home/yourusername/myproject/myproject/img/favicon.ico;
        expires 3d;
     }

    location /static/ {
        root /home/yourusername/myproject/myproject/;
        expires 3d;
     }

    location / {
        proxy_pass_header Server;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 10;
        proxy_pass http://localhost:8150/;
     }
 }

, . , , ( - ), static files versioning ( , query string, , , query string ). , , () CSS JS — , django-assets .

haproxy:

global
    maxconn 10000 # Total Max Connections. This is dependent on ulimit
    nbproc 2

defaults
    mode http
    option redispatch
    maxconn 2000
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000
    option httpclose

frontend all 0.0.0.0:80
    timeout client 86400000

    acl is_chat hdr_beg(host) -i chat
    use_backend socket_backend if is_chat

    default_backend www_backend

backend www_backend
    option forwardfor # This sets X-Forwarded-For
    timeout server 30000
    timeout connect 4000
    server server1 localhost:8100

backend socket_backend
    balance roundrobin
    option forwardfor # This sets X-Forwarded-For
    no option httpclose # To match the `Connection` header for the websocket protocol rev.  76
    option http-server-close
    option http-pretend-keepalive
    timeout queue 5000
    timeout server 86400000
    timeout connect 86400000
    server server1 localhost:8000 weight 1 maxconn 5000 check
    server server2 localhost:8001 weight 1 maxconn 5000 check
    server server2 localhost:8002 weight 1 maxconn 5000 check
    server server2 localhost:8003 weight 1 maxconn 5000 check

, , Host chat, localhost:8000, localhost:8001, localhost:8002 localhost:8003, localhost:8100.

haproxy 80- IPv4-. IPv6 ( , /), frontend bind IPv6- 80- .

, IPv6- (, , AAAA- DNS), . , . — IPv6, DNS whitelisting . , , — IPv6, 1—2 ( ).

, Tornado- , — , ( 80 443).

haproxy , backend Host, Upgrade — , Django, Tornado ( ), WS- Tornado-, — Django-.

, . , , , ( Django Tornado ), , . ! .

Source: https://habr.com/ru/post/160123/

All Articles

Tornado WebSocket Chat for your Django project

Django setup

Chat (Django)

Chat (Tornado)

Deployment

More articles: