Distributed video encoding

I think the majority of users at least once had to encode the video. Maybe you wanted to watch the latest episode of your favorite TV show on the subway, your smartphone or other portable device didn’t want to play this codec, or SmartTV refused to decode the sound in a video file, or you got stuck in 2004 and transcoded H.264 to MPEG-4 for your old dvd player. Or, for example, they squeezed an incredible-sized video taken with a camera into something more or less decent in size. Surely you have noticed that this is not the fastest process.

A huge amount of software has been made for video encoding, starting from programs like “click the cross to encode” and ending with software that accepts AviSynth scripts as input. However, programs that support at least some coding distribution can be counted on the fingers of one hand.

What generally should be understood as distributed video coding? In my view, there are 2 different tasks:

1. Encoding a large number of video files on multiple computers simultaneously
2. Encoding one video file in parts on multiple computers simultaneously

The first task involves a large number of video files, much more than computers on which they can be encoded, and is quite easy to automate using CLI encoders, such as HandBrake or FFmpeg, and any automation tools that support distribution, such as GNU Parallel or PPSS ( By the way, few people know about him, I recommend!).
')
The second task implies the presence of a small number of files and a large number of computers, and it is somewhat more difficult: we need, firstly, to split the file into small parts, secondly, to recode them, and thirdly, to assemble them back.

Usually, I need to encode 1-2 10-bit video files encoded with the H.264 codec into 8-bit, with a medium , slow or slower preset . Typically, one minute of video takes 6-8 minutes of coding. I would like faster.

Glossary

GOP (Group of Pictures) - literally, a group of images. Some frames from one I-frame (key frame) to another, not including the last one.

Worker is a program that does some work (usually the most expensive) in a distributed system. In our case - video encoding.

Turnkey solutions

I spent several days looking for live projects for the distributed coding of both tasks, and this is what I found:
1. MediaCoder
MediaCoder is a fairly advanced and convenient harvester, balancing well between the number of settings and ease of use. However, the distributed encoding in it is worse than nowhere implemented: the video is decoded locally, and the worker is sent an uncompressed decoded frame. Recommend gigabit network, so that through the Internet it will work incredibly slowly. The server works only under Windows (and also in Wine), the worker works under Windows and Linux. Only H.264 and VP8 are supported.

2. dvd :: rip
The distribution methods in this project, unfortunately, are no better than using conventional means of distribution. You will need to raise the SSH server on each machine and the NFS server on the file machine. Can not encode a single file on multiple computers.

3. Ripbot264
The best I've seen. It cuts files correctly (by GOP), is conveniently configured, but, unfortunately, it uses the Windows file sharing tools, which virtually eliminates work outside the same network (the program requires that all computers be in the same workgroup). And it works only in Windows.

Build your bike

Unfortunately, I didn’t find anything tolerable under Linux and had a great idea to make my own. What requirements did I put forward for a distributed video coding system?

• The system should be able to encode one file into several workers (cutting the file by GOP and then merging back)
• As many decoders, encoders and containers as possible (at least mkv and mp4) should be supported
• If possible, avoid creating temporary files and minimize RAM consumption on workers.

For video encoding, I decided to use FFmpeg , and for cutting and pasting the mkvmerge files, and, accordingly, the matroska container (because during testing, FFmpeg got out a regression in mkv-muxer).
I rewrote the project 6 times. What ideas were discarded:

• Use Job Server ( Gearman , Beanstalkd ) and Message Broker ( RabbitMQ ) to transfer chunks of video for encoding inside Job. This, of course, is very convenient, but I could not afford to keep pieces of the source and recoded file in memory. In addition, you first had to get the video completely, then recode it, and only then send it.
• Using RPC for the same reason
• Using an HTTP server to upload and receive files

And everything is back to the sockets. Indeed, it is better to come up with something better than sockets - a socket can be fed both to stdin FFmpeg and to stdout, and no temporary files will be created, and the RAM will be spent to a minimum, and the encoded video will be uploaded directly to the server.

And I did it.
github.com/ValdikSS/distvidc - DistVIDc (like distcc , for video only)
I used Rage Driven Development, and notably relaxed when I received the first working version and the first commit.

How does it work?

There are three scripts - server, client and worker. The server waits for the file and the encoding parameters from the client, cuts it into pieces, distributes it between the worker, gives the parts and accepts the reconverted pieces, collects the pieces into a file. Worker, respectively, connects to the server, waits for the command and encodes the pieces.

Problems:

• Variable frame rate (VFR) video is likely to be incorrectly encoded and out of sync.
• The coding efficiency is slightly lower due to the fact that at the end of the cut piece of the GOP may be less than it could be without cutting

Positive sides:

• You can submit to the input and receive at the output all that is supported by the FFmpeg project . Theora in ogg in VP8 in webm? No problem. God forgotten msmpeg2 in avi in ​​HEVC in mkv? Yes, easily!
• Efficient work worker. Only about 200KB of RAM is consumed!

DistVIDc will work both under Linux and under Windows (at the moment, the worker works only under Linux). I will be glad to anyone who is interested in the project.