Delivery of video content to users

[Part I. Delivery of video content] [Part II. CDN do it yourself]

What is “content” for video hosting? Firstly, video hosting content is simply a video, which is a collection of files in various formats, in particular, in FLV format for viewing by the user through Flash Player. These files are static, video hosting when a user uploads a video, converts it into all required formats with the required bit rate. The storage of such content is the storage of regular files, only of rather large size. Content delivery is essentially the organization of downloading files.
Secondly, the content of the video hosting is a “live” stream or broadcast. Broadcasts are not recorded to disk, they are not converted, streams are distributed to clients taking into account bandwidth (there is a pass of packets if the client’s channel is not enough to receive the stream in full quality). Content delivery in this situation is the distribution of the stream to a large number of connected users (thousands of viewers).

In addition to the content delivery task as such (correctness, work under load, etc.), the problem of “approaching” the content to the user is relevant. It is necessary to organize the delivery of content so that the user receives video or broadcasting from a server that is located close to it in terms of bandwidth of the network channels, as well as in terms of the cost of traffic for the user.

This article, written on the materials of the report at the conference “RIT: High loads” -2008, will tell about one of the possible approaches to solving the tasks set. The story will be based on our experience in the design, development and support of Smotri.Som video hosting, which is the largest Runet video hosting service today. The described approaches can be applied in areas with similar characteristics of content: a sufficiently large amount of files, many files, various types of broadcasts, etc.
')
The article will consist of two parts: the first part will deal with the organization of file storage for video files and about the general aspects of the organization of broadcasting. The second part will present the method of organizing a CDN ( Content Delivery Network ), that is, a method of “approaching” content to the end user, as well as applying this approach to the delivery of static files (video files) and to streaming (broadcasting).

Video: file storage organization

Video files

Video hosting in the course of its operation receives video in various formats, which is uploaded by users. In addition to storing the original video file, video hosting converts a file into a number of formats: FLV , 3GP , MP4 , etc. All these files must be stored on file servers, as well as given to users. Consider first the issue of storing such files

What are these files? According to our calculations, an average of 250 KB of disk space is required to store one second of video (meaning 250Kb / s for all video formats taken together), while the average video length is about 4 minutes. It is easy to calculate that for storing 1 million video files, we need 60 TB of disk space, which is already a rather impressive figure. This storage capacity must be efficiently managed, and also this content must be distributed to users. In addition to directly video files, we will have to store and deliver frames cut from video, which in our case will be 15 pieces: 5 frames cut from different parts of the video, each of which is presented in three different sizes (to display information about the video in blocks of different formats).

Access to the file server via WebDAV

Directly for file storage is the responsibility of the file server, in which there is a fairly well built disk subsystem in terms of reliability and speed of access (RAID). Requests to change files (creation of new files, deletion of old ones, etc.) come from another server group: from the muzzles (frontend) and from the recoders. Muzzles are normal servers that serve the vast majority of HTTP requests both directly to the site itself and to its API. Transcoders accept requests from users to download the source video files and convert them. So, on the face or transcoder an order is created for the operation with files on the file server, since the file system is not local; some network access to the file server is needed.



The simplest solution is NFS , which can save for the site code a “feel” that the files are located locally, while taking on all the difficulties of performing remote operations through the network. However, NFS can be extremely unreliable when the network connection between the muzzle and the file server drops, and also because there are dozens of muzzles and file servers, the number of NFS cross-mounts becomes too large.

Another solution in this situation could be WebDAV. WebDAV is an extension of HTTP, which, by the way, initially assumed that the content of the World Wide Web is not static, but changeable. WebDAV takes another step forward by adding full-fledged capabilities in HTTP to remotely modify, create, delete files, directories, get information about files, etc.

Within the cluster of video hosting servers, access to file servers is based on the WebDAV protocol. Downloading files (including playing video in Flash Player) comes directly from the file server, which allows you to download network channels more evenly, excluding the “extra” intermediate links.

Choosing a file server from a cluster

Suppose we have a cluster of file servers, everything is set up, ready to give files, and also to receive new files via WebDAV. A video has been uploaded. Which file server to choose to store the next file? The selection can be made, for example, based on the following characteristics:

• free space / server volume;
• server load (how to measure?);
• random selection.

An example of a bad strategy is the desire to fill all servers so that the percentage of occupied space on the server (or just the amount of free space) is equal among all servers in the cluster. With this approach, the servers are evenly filled, but in practice the project starts from several file servers, then it is added again and again, and it turns out that when you add a new server to the cluster, all new files are added to this server (because most free space). And new files are often the most popular in a given time period, so the load on new file servers in this configuration increases many times over.

Such a strategy is much more successful: among all file servers, those are selected whose fill level has not reached a certain critical level (that is, those on which there is still free space), and then a random selection is made among them (perhaps, the choice with taking into account the weight of the server). With this strategy, popular videos are evenly distributed among a larger number of servers, which allows you to more evenly distribute the load on downloading content.

Required software

The approach described above requires simple and common software: any “fast” HTTP server, for example, nginx , lighttpd , etc., is suitable for downloading files. At the same time, in order to provide FLV upload to the player from any time stamp (the so-called “streaming”), we need a very simple extension, which is present today in both nginx and lighttpd (flv-streaming). Apache httpd etc. is suitable as a WebDAV server. To access files via WebDAV, you can use any WebDAV client that is available in one form or another in almost all programming languages. For example, in PHP it is implemented as a PEAR class (which has to be “doped” to ensure reasonable performance). WebDAV does not provide functionality to obtain free disk space, but it is easy to get around using the simplest cron script, which redirects df command output to a file in the root of the file server, and this file can already be downloaded via HTTP from any muzzle.

Cross Backup

How to ensure reliable storage of 60 TB of data? (Provided that this figure will grow?) Traditional options with backup to the central server are no longer suitable, since Such storage capacity on one server is difficult to ensure, and the reliability of such a scheme is low. No RAID level within a single server can guarantee reliability, since Both hardware and software failures are possible, leading to data loss. A simple and fairly reliable way is cross-backup: each of the file servers is filled in half, and then the contents of the first server back up to the second, and vice versa. Thus, if a failure occurs on one of the servers, the full content of both servers will remain on the second.



To implement this approach, a simple control module is required, as well as rsync. When using cross-backup, it is interesting that by the same information about the free and occupied space on the file server it is impossible to judge its fullness, since after synchronization of the backup part, an arbitrary increase in the storage volume is possible (if the breakdown is not known in advance: how much space is occupied by the data itself, and how much is the backup of another server).

Broadcast: Relay

A detailed story about the organization of broadcasts and our broadcast server has already been presented at the RIT-2008 conference, so here I will not repeat myself and go into details, I’ll dwell only on the main aspects.

Broadcasting consists of video and audio streams that are encoded on the client side using Flash Player, then using the RTMP protocol are sent to the broadcast server, which distributes these RTMP streams to all viewers connected to the broadcast. The main problem here is that the stream is distributed to customers, albeit without recoding (in quality and with a bitrate specified by the broadcast author), but the number of clients connected to the broadcast can be thousands of people. At the same time, a modified stream can be delivered to each client in accordance with the bandwidth of each client (packet drop in the stream). In addition, in the RTMP stream, in addition to the broadcast stream itself, there are remote procedure calls (server-client or client-server), as well as the state of shared objects among all clients connected to the broadcast.



One server can not always cope with the task of distributing the stream of broadcasting, so we resort to relaying: the broadcast author connects to the primary broadcast server, to which other broadcast servers join the clients, which in turn distribute the stream to the final viewers. With this scheme it is possible to evenly distribute the load between the servers broadcasting (the number of clients on each server is approximately the same).

The next part will be about creating your own CDN. To be continued...