"We continue our report!"
Nikolai Nikolaevich Ozerov

The legendary TV commentator Nikolai Ozerov, who would have turned 92 this year, has turned sports broadcasts into shows that attract millions of sports fans to TV screens. The technologies have since gone far enough, but the essence remains the same - not being able, due to various circumstances, to personally attend an interesting sporting event, you can follow them through telecommunication technologies.

Today, MegaFon went to meet sports fans and avid fans by launching a special telecommunications application, LOOK +.
')


Habrahabr has already published several articles on this application:

1. Interview-story about the application of the project manager Andrei Kushnarev.
2. A colorful report on the opening of the Olympics in Sochi and the stress test of the entire network .
3. In May, MegaLabs conducted work on the bugs and told this to all habra residents.

But time does not stand still - the application develops and conquers its users. The number of downloads has already exceeded 500 thousand half a million only for the Android platform! (Marketers joke: "half a million" sounds more solid, but this is really a cause for pride). And there is another version for iOS , where they crossed the threshold of 100 thousand active and, we hope, grateful users.



Today we will tell you what happened new in the past six months and reveal some secrets that are hiding inside, on the other side of the communication channel. But first of all, I would like to reveal the “secret” of the phrase “telecommunication application”.

The fact is that for commercial exploitation of such a service it is absolutely not enough just to attach an application - this is a rather complicated information and communication system. It starts at the stadium (hockey arena), produces several stages of video signal processing: the signal is saturated with additional information, the finished video stream is distributed over a huge Russian territory, and only then the user will be able to follow the game of his favorite team with enthusiasm on his gadget.

At the same time, the development and support team, during a period of just over one year of hard work on the service, was able to formulate exactly what advantages the “LOOK +” application has over “regular” television broadcasting:

Firstly , sporting events are most interesting in real-time mode “here and now” without spoilers and disclosing the intrigue of the game. Television, for the sake of justice, is also able, but in linear TV there is a rigidly defined broadcasting network within the channel, and interesting events can take place simultaneously and it is not always convenient for the viewer.

Secondly , unlike TV-series, a sports fan needs more information - the schedule of matches, statistics of teams and players, comments and news. “Normal TV” such information binding is technologically unavailable.

Thirdly , a lot depends on the quality of the picture - it is much more pleasant to observe the event closer and with details. For example, in HD-quality broadcasting, the probability of seeing the puck and the direction of its flight is much higher. And therefore - more interesting to watch.

Fourthly , unlike “linear television”, the application “LOOK +” allows you not only to record and then reconsider the most exciting matches, but also to repeat the most interesting moments that were selected by editors and other users. This in itself increases the value of content for fans who like this feature.

Fifth , the application ensures the availability of video anywhere without freezing and interruptions. We tried very hard to achieve a quality in which our users will not miss a single exciting moment.

Five points are usually enough to indicate the advantages of our product “above ordinary powder”, but we also have sixth: we can now select commentators for the match and point of view cameras. This is a rather experimental function with not yet a clear economic effect, and it is available only for some especially popular matches, but we are working on debugging this function throughout the content production chain.

Let's look at the scheme of the service more:

How it works

"Sorry, technical problems outside the USSR"

This distracting phrase was used by Nikolai Ozerov when fights began during hockey matches. The camera switched to a neutral fan, and the commentator talked about events in the world of sports. In one of the reports, when the “duty phrase” was again used, Nikolai Nikolayevich even managed to tell about foreign tours of the Soviet ballet.

But we are not going to talk about ballet, but we will publish the functional scheme of the “LOOK +” service:

Content is king

The fundamental difference between “LOOK +” from the portal with Internet TV lies in the content itself and the way it is delivered to the viewer. In the application, we do not broadcast the content of any finished television channel, but generate a signal from the very place of the event. And this is a rather complicated production process.

Content is created and sent to the viewer "LOOK +" in several stages:

1. First, the “picture” is created from several video streams that come from all the cameras in the stadium or the venue of the sporting event.
2. Next, all the signals from the cameras come to a mobile television station (PTS). In the TCP is the director of the broadcast, which forms the so-called output signal. It is this “live” signal that the viewer ultimately sees.
3. Video streams from the PTS via special communication channels (as a rule, these are VPN channels with very strict SLA requirements) arrive at the distribution hub, from which in most cases the signal goes to our hardware room for editing, overlaying the broadcast of graphics, adding special effects and commentary sound. Sometimes the installation of the commentator and video effects can occur in the TCP.
4. From the central hardware processed and ready for consumption video stream returns to the distribution node. At this site, there is a "distribution of the signal" to customers - TV channels and the actual platform "LOOK +"
5. And only after this, users of the “LOOK +” application can enjoy matches with special graphics and commentators.

The entire production process and the signal delivery process are divided between specialized partners: television companies make the shooting and the rest of the production, and the technology partners produce the signal delivery. Between the companies established round-the-clock interaction and control of each broadcast. We control each match and check that all declared signals follow the appropriate lines and with the appropriate quality.

For security reasons, we can not publish photos of our mobile television stations - so as not to attract the attention of potential vandals who may harm the broadcast. In general, the PTS looks like an ordinary bus or truck-wagon, where all the most interesting is inside. Approximately this type has the TCP outside and inside:

Video signal processing

We have already mentioned above that the video signal is subjected to special processing - digital broadcasting has a number of undeniable advantages over traditional analogue methods of transmitting a TV signal: for example, we can produce a picture of the quality that the viewer can and will want to receive. If the data channels are not very good for the user at this particular moment (for example, the 3G in the center of the megalopolis in the CNN is utilized quite tightly), or the user's end device cannot process video in HD quality, the user is automatically prompted to “pick up” the SD quality. And the opposite situation - if the subscriber has a wide enough channel (LTE, for example, or Wi-Fi at home) and has a modern tablet with a powerful processor, then why not watch the broadcast in maximum quality?

As a basic standard, our platform uses the adaptive broadcast standard Apple HLS ( HTTP Live Streaming ). This protocol allows broadcasting to most of the available terminals (subscriber devices). In our case, it was necessary to ensure the availability of the service on Apple iOS and Android of different versions. To ensure the performance of HLS in conjunction with the content protection system Verimatrix had to use a third-party player from NEXStreaming. He was chosen on the basis of the technical capabilities of ensuring the specified characteristics of the player.

Media business

Sporting events are a rather costly business that should be economically viable. Content protection and use of DRM systems ( Russian version ) is the most important of the requirements of the many partners involved in the broadcast creation chain.

The use of copyright protection systems does not allow users to make pirated relays of matches. Mainly due to the fact that they have nowhere to do it - the “LOOK +” platform independently receives a signal at the point of its formation, then it is processed (we compress, we form video streams of different quality, we divide into blocks, we encrypt) and is placed on the CDN.

To encrypt content and verify legitimate users, the Platform “LOOK +” uses the Verimatrix solution. It was chosen for several reasons: firstly, Verimatrix is ​​the global leader in content protection. And secondly, Verimatrix solution provides content protection for our second MegaFonTV television / video project. The use of Verimatrix tools allows you to quickly pass a platform security audit with most rights holders, which was very important due to limited development time.

The principle of operation of this DRM system is not the subject of this article, but if very briefly, it lies in the fact that each copy of the “LOOK +” application implements a unique “access key” that is generated when the application is installed on the user's device. Then everything is simple - if a user without a key or a fake key accesses the HLS servers, the request will be rejected. Certainly, the algorithm of verification of the “access key” is not disclosed, but an inquisitive habra reader can, for example, refer to the original source and search engines.

Signal delivery

The process does not stop to form a signal, encrypt it and authorize the user. One of the most important criteria for the quality of a video signal is the quickest delivery to the user without delay and lags. To solve this problem, a whole network of specialized CDN servers is used.

Most of Habr's readers are probably familiar with the Content Delivery Network concept, therefore, very briefly and concisely. If readers are interested in the specifics of the solutions of the MegaFon’s own CDN network, we will definitely write more in detail.

And the federal operator owns a CDN. This solution allows you to distribute the load on the “LOOK +” service platform (and the already mentioned “MegaFonTV”) with large user viewing peaks - the more popular the match, the more users. In special cases, the network will “pull” hundreds of thousands of subscribers who are simultaneously watching videos, which was confirmed by the practice of Sochi-2014. It also solves the issue of signal delivery to the regions of Russia. At the same time, the use of this CDN does not affect the availability of content for subscribers of other operators, since it does not depend on the access point to the Internet on the user's side. We are constantly improving our service and conducting tests with new CDN partners, but in the commercial operation “LOOK +” we use our own CDN.

And some more details.

The application and its support infrastructure are constantly being improved. The first version of the application (for the Olympic Games) was developed based on a ready-made server solution from a foreign partner. Such a decision was justified, since it was necessary to launch in a short time. But the application did not have 100% of the functionality that we wanted to implement. As a result, by September of this year, we have completely reworked both the application itself, and many elements of the server-side infrastructure. The developer of the new version was the team MC2Software, which in a short time was able to implement the new functionality of the application for launch under the KHL, as well as maintain access to the archives of sporting events launched earlier.

We intend to continue to develop, improve the quality and complement the functionality of the service. One of the goals of writing this article is to collect ideas and needs Habrazhiteli - feel free to comment. We will not only carefully read the entire feedback, but also draw up a list of new functions and features of the application. And be sure to write about their implementation.

Server magic

The description of the internal kitchen "LOOK +" does not end there - gradually we get to the most interesting - "what's inside" and some organizational details.

Content Sources

Sports fans are not housewives who just watch live pictures on the screen - these are active fans who are very interested in using additional information: sports statistics, results of matches, teams, player information and a thousand small but important informational messages. .

“LOOK +” is radically different from the majority of implemented Internet broadcasting projects - all the necessary sports data is carefully collected, processed and made available to users as a final visual (?) Product. Of course, as a technology company, MegaFon is not a recognized expert in the field of sporting events — primary data are collected, specially trained people from our supplying partners handle them. And, as it turned out, the choice of the supplier of sports information is a non-trivial task, which is also superimposed on the technological limitations of software solutions.

First, the selection of the supplier of sports data was carried out according to the criteria of cost, type and quality of API, quality of documentation, data update speed, etc. As a result, Opta Sport (Perform Group) was chosen, the company - the world leader in collecting and analyzing information about sports events in real time.

With the active participation of our partners, we have developed data structures in such a way that, on the one hand, they contain all the necessary hockey (we are talking primarily about KHL) statistics, and on the other, they are quite flexible and extensible for other seasons and sports.

At the stage of practical implementation of the data import scheme, debugging, launching and writing dozens of letters to the supplier, which began with the phrase: “here you are wrong, correct, please,” there was a problem of flexible adaptation and editing data “on the fly”. And “LOOK +” got its own system for preparing and publishing content, as well as a professional editorial department.

There is an interesting point. The data that we get from Opta, the editor can partially correct. For example, many player names come to us in a transliterated form. The editor can write a normal name, and it will not be rubbed by the next import from Opta, but, for example, the player’s statistics will be updated.

Pushing through

The next interesting point in server-side magic is the mechanism for retrieving data updates. The fact is that unlike “linear TV”, where ordinary streaming is conducted, the “LOOK +” platform generates a huge number of notifications for the application - sports statistics, commentators, including text, events on the sports field - everything described in the previous section of the article. All these events are forwarded by the server side to the application in “push” mode. The application may not know when the event will occur, but should immediately respond to it in the event of an event.

Typically, most applications receive events by periodically sending http-GET requests — first a request is sent to the server, the response is received, then the screen is filled with them. Such a mechanism is easy to use and programming, but at the same time a large amount of meaningless traffic arises on the Web — the application makes a request, and the server responds to “nothing happened”. And so several times a second. Imagine that hundreds of thousands of users behave in this way, and the scale of the disaster appears to be in full growth.

We went another way: the application contains a local copy of the database, which is updated at startup and in the background, and in the active state, each copy receives updates with minimal delay. For example, with a good connection, video moments appear in the player within a few milliseconds after they are created by the editor in the content management system. This mechanism also allows you to use part of the application functionality even when the communication session is broken, which is not uncommon when using mobile Internet access.

The mechanism, called Server-sent events , forces the application to receive all elements of the news feed (video moments, polls, news, etc.), game moments and comments from the timeline, lists of matches, teams and players, other information as it arrives.

At the same time, statistics for internal use is collected in a “normal” manner (by a request to the server) and is not stored locally, in order not to significantly increase the size of the local database.

Go

The use of the server-sent events mechanism imposes additional requirements on the toolkit, and we came to the conclusion that solving not the most standard tasks requires similar tools. And we chose the language Go.

On Habré there is a whole hub dedicated to this wonderful tool, and a link to the source is given according to the tradition and rules of good taste.

The main reason for choosing this solution is the mechanism for receiving updates, which keeps a constant open connection. Making it on, for example, Ruby or Python would be difficult and non-idiomatic, and on Java it would take longer, especially considering that we do not have Java developers. :)

The use of language has significantly reduced development time, despite the fact that we have learned the language "in the process." You can install the result on your smartphone and test it. We also made a component for sending push notifications, which is able to send 7000 ios-pushes per second to one connection.

Instead of an epilogue

“Now the State Flag of the Soviet Union will be sounded!”
Nikolay Ozerov

As already mentioned, the service “LOOK +” has been operating since February, from the Olympiad itself. According to the results of feedback received from users, “work on the bugs” was done and in September the next release of the product was launched. The important difference between the current version and the one released for the Olympics is the content itself and the events generated.

Previously, for each new event we had to update the application in the app stores (App Store and Google Play), now we can add sporting events without updating the assembly. This was a major business requirement.

According to application statistics, we currently have about 1.3 million downloads, but slightly less active users. At the same time, the permanent active audience of the service with content from the KHL alone is more than 200 thousand unique users per month. The most active city by pageviews is Novosibirsk, where we are watched the most.

Development plans, as usual - ambitious.

Soon we will make the "multi-chamber" function for some KHL matches. Basically, the function is ready in the application itself, but there are some problems of organizational nature and usability. “Multi-camera” should allow the user, when watching a match, to choose one of several cameras that are located in the stadium, which will allow the fan to change his point of view right during the match, and then review certain moments from different angles.

In addition, there is a goal to add elements of gamification to the application. The fans are the people involved, and we want to build a dialogue with them in the form of additional exciting events. For example, to hold more interesting contests during the month, and at the end to determine who was the most active or collected the most points, etc. We are also thinking about adding content to UGC. At the same time, we understand that to launch this functionality, you need not so much resource (financial, human, time) costs as it is important to fill ideas with content and common sense. The main thing is not the implementation of “funny features”, but the relevance of the functionality by our audience, its actual use and necessity. We very much hope for your help and support and will closely follow the comments.