Text online translation with a large load

A large role in the life of our site is played by the online presentations of Apple presentations. In addition to the text-graphic broadcast, which make many Russian-language resources, we offer our readers to lead the discussion in the chat, as well as conduct a live audio broadcast with comments on everything that happens. As in previous times, Kira Tverskaya (Echo of Moscow) and Eldar Murtazin helped our regular hosts. Such a lively format allows us to hear the reaction of the presenters in real time, which you will agree, is quite interesting.

Last time, at the presentation of the iPhone 4S, we blundered a little and could not adequately prepare for the influx of visitors (by the way, there were about 3,500 customers). The main mistake was that the translation page was the output of the Wordpress plug-in, so each time, not only the PHP interpreter itself was twitching, but the entire Wordpress system was twitching. After the broadcast, it became obvious that such a shameful scheme must be disposed of.
')
Rumor has it that in March, Apple will present a new version of its iPad 3 tablet. We were lucky, and on January 19 we were able to run in a new broadcast scheme at a much lower load than it would be in March. For obvious reasons, there were far fewer people interested in Apple’s educational presentation.

We had a live broadcast on all fronts - a text broadcast with photos of the event, an audio stream and a chat in which people could discuss what was happening with each other. Next, I will talk about the technology with which the text translation was organized.

Here is a general distribution scheme for text content.

The key role in this scheme is played by distribution servers capable of supporting several thousand long-lived HTTP connections and almost instantly distributing new data to connected clients. Their work is based on the Dklab Realplexor Comet server, about which the developer DmitryKoterov himself wrote in Habré .

An admin is a simple web interface written in PHP. Allows you to conduct several broadcasts, and add, delete and edit individual messages of the broadcast tape in them. The principle of the admin is as follows:

admin enters a new message for broadcast stream
the message is added to the admin database
The first 10 messages are pulled out of the database and the index.htm file is generated.
all messages are pulled out of the database and the full.htm file is generated
The resulting index.htm and full.htm files are uploaded to CDN CloudStorage using the OpenStack API

A new message is sent for distribution to all Comet servers, using approximately such POST requests:

 POST / HTTP / 1.1
 Host: comet1.live.appleinsider.ru
 Content-Length: 30
 X-Realplexor: identifier = ai

 {"time": "07:36", "text": "test"}

Now, as it looks from the client, i.e. listener / reader online broadcast.

The client enters the address of live.appleinsider.ru and his cached HTML page loads up relatively quickly. Let me remind you that CloudStorage is a cloud storage from Clodo, which has a wide channel and high speed content delivery.

In this “seed” there is a Javascript code that selects a random number from 1 to 4 and makes a connection to one of the Comet servers comet1 — comet4:

<script type="text/javascript"> var comet = new Dklab_Realplexor("http://comet"+(1+Math.floor(Math.random()*4))+".live.appleinsider.ru"); comet.subscribe("ai", receive_post); comet.execute(); function receive_post (msg, id) { if(msg){ $("#onair .new").removeClass("new"); html = "<li class='new'><span class='date'>"+msg.time+"</span> "+msg.text+"</li>"; $("#onair").prepend(html); } $("#chat").append(msg.text); document.title = "* *"; setTimeout( function(){document.title = "AppleInsider.ru";}, 5000); } </script>

In general, they could be smashed into different data centers, but in our case we decided not to bother and just load the 4 processor cores. Each dklab_realplexor process occupied one core, and the core balancing was performed by nginx. When new messages appeared on one of the distributors, they were almost instantly delivered to customers who are connected to it.

And finally, a small technical report on the broadcast:

  Date of broadcast: January 19
 AppleInsider.ru audio broadcasting time: 18: 30-20: 00
 Apple presentation time: 19: 00-20: 00
 Audio stream listened: a maximum of 750 people
 Text translation read: a maximum of 420 people
 In the chat sat: a maximum of 140 people