Streaming and processing video through node.js + php and ffmpeg - part one

I work in an outsourcing company and somehow there was the task of downloading videos with the possibility of subsequent processing for the internal needs of the application: resize to the desired size, converting to the desired format, pulling audio tracks (if any), storyboard video. At the end, the results should be stored in the cloud storage for later use in the online editor. Requirements: scalability, unlimited video size, speed, cross-browser compatibility, visibility.

Since the topic is very extensive, I will divide it into sections:

1. Common problems, the nuances that had to face
2. Video uploading (I’m probably not going to stop on this topic, since it has already been raised in this and this post.
3. Video processing.
4. Saving in the cloud storage.

Part one

1. Cross-browser compatibility
In order to ensure this requirement, HTML5 file uploading could not be used (it is not implemented in all browsers). Therefore, we went by creating the usual HTML4 form that submitted to the server with node.js. In order to get the current progress of video loading / processing, AJAX requests were created. The form included a randomly generated id string (for example, 32 characters). Node.js tied to it the information on the current file and with each request for the state it transmitted the following JSON:
')

{ "fileid":"uwrd28a9v71j444d260c55hkj6uli06j", //  "name":"test.FLV", //  "progress":"7.80", //   "status":"uploading", //  "audio":10.02, //   "frame":301, //  ,   "videoProgress":"7.80", //   "audioProgress":"3.73", //   "frameProgress":"3.73", //   "messages":[], // ,      "expectedBytes":5389574, //   "uploadedBytes":420139, //   "bps":91773.48186981214, //  / "uploadEstimated":54, //     "videoEstimated":42, //      "audioEstimated":75, //      "frameEstimated":71 //     } 

Based on this, the user sees the status of the download.

2. User authorization
Since the frontend in PHP (user authorization is happening there), it was necessary to somehow transfer the session data to the node.js server. For these purposes memcached was used. About changing sessions in PHP can be read here . The point is simple: sessions are saved in memcached, while loading, a session_id is transmitted via the form, which is read by node.js. Further node.js addresses to memcached in search of the corresponding session, takes user_id, etc. There is one important point: the session_id in the form itself must be set first, because the POST request passes the parameters in the order in which they appear in HTML. That is, if we do this:

 <form method="post" action="http://nodeserver.com/"> <input type="file" name="video"/> <input type="hidden" name="session_id" value="session_id"/> <input type="submit" name="submit"/> </form> 

In this case, the file itself first comes to node.js, and then session_id, which is not good. After all, we first need to know who the file comes from and cancel it in case the user is not authorized. If we swap file and session_id in the example above, we first get the session, slow down the file download, and wait for the server to check if the user is OK and only then continue the download. You can pause requests in node.js.

3. Stream video processing
The problem is that not all videos can be processed immediately. For example, according to the mp4 format specification, metadata comes at the end of the file. But we need to immediately know what size the video has, what tracks there are in the file, the duration, etc. Plus, some video formats need to be converted during transcoding in order to be able to access different parts of the source file. Based on this, there are two cases with two options each:

1. At the beginning of the download, when we pull out the file information using the ffmpeg command, using node.js:

 var spawn = require('child_process').spawn; var ffmpeg = spawn('ffmpeg', ['-i', '-']); var ffmpeg_stdout = '' ffmpeg.stderr.on('data', function(buffer) { ffmpeg_stdout += buffer; //     }); // file -     file.on('data', function(buffer) { ffmpeg.stdin.write(buffer); }); 

In the variable ffmpeg_stdout we get all the tracks that are contained in the file or an error. If there is an error, then ffmpeg cannot process the file by stream. In this case, we warn the user about it and load the entire file at once, and only after that we perform the necessary operations.

2. In this case, the speed is important, so we are trying to start the storyboard as soon as the video has just started loading. But if the format does not support stream processing, then you have to wait for the moment of full download.
It turns out this algorithm: it is possible to extract the metadata, then we try to process the video on the fly, and if it does, we are happy, in all other cases we are waiting for the full load. In practice, only a small amount of video is required to download completely to begin processing.

For today perhaps everything, tomorrow morning to work.

I would like to finish on a positive note: the possibilities are endless, you can distribute the load on several servers, show the results on the go, for example, pull the last ready frame from the video. It works all on Amazon EC2. By speed, I can say that the download speed was not possible to check (the channel is weak), but not less than 10 Mbit, I think for sure. And on the server load: Middle server alone, one video pulls somewhere around 2 Mbps of video, that is, the download has gone far ahead, and the processing does not keep up. The longest time is resize. The storyboard and pulling audio track relatively quickly.

If you like the post, I will describe the implementation and mechanism of work in more detail in the following times.

Screenshots of what happened. Immediately make a reservation, this is not yet a working version, so do not judge strictly: