Making a home farm for video rendering.
In this article I will talk about my experience in creating a separate server (the “farm” is painfully loudly said) for rendering video at home.
As you know, rendering of video and three-dimensional images takes a lot of time and requires a lot of computer resources. My memories are still fresh: as a student, I put a humming computer with a running movie build program at night, and in the morning it turned out that either there was not enough space on my hard drive, or I forgot to add something to the video and I had to start all over again. Now real professionals do this operation remotely. For example, during the filming of Avatar, director James Cameron made an order for himself to create a whole data center, in which scenes were rendered. I'm still not so cool to build my own separate data center, but the idea that this operation could be run separately on another server did not give me rest for a long time.
')
The second problem is the installation of the video itself. Despite the fact that modern computers are becoming more powerful, the development of video cameras also does not stand still and the resulting video files are becoming harder and harder. And, as a result, it becomes problematic to handle them. When you mount a long film, the preview of the track with superimposed filters and transitions begins to load the processor and the image starts to lag, making the creative installation process tedious. From what I have tried, the most productive was iMovie, which is preinstalled on all Macs. Even a 45-minute film could be edited without any problems on a relatively low-power macbook. You could apply any filter and see the result in the preview window without any lags. So the poppy owners are lucky here. One minus in iMovie: during operation, it begins to consume an inordinate amount of disk space. Apparently, this is due to aggressive caching for editing and previewing.
In this article I will describe my experience as I tried to solve these two problems of editing and rendering with the example of a home video clip.
If you compare with Windows or MacOS, for Linux there are usually not so many desktop applications, but the choice of video editors turned out to be surprisingly wide. Having played with several options, I stopped at Kdenlive for working with small videos. When comparing with other programs, there is a good set of functions, but in this article we are only interested in one, but a nice feature: when mixing an already finished movie, the program offers two options - as usual, generate to a file or generate a script.
At the same time, it will simply create a regular shell script, which you can then run from the command line. Including on another computer. What am I getting at? Yes, just so you can reduce the project on a remote server. To do this, you can use the clouds, your server, or even a powerful work computer, which remains constantly turned on in the office (but I, of course, did not say that). The plan is simple: we reduce the project, generate the script, transfer the entire project to the server, start the process remotely.
But it sounds too easy for real bison. Add some conditions to the task. I have already complained that installing original video files is a task that requires a lot of resources. While working, we view the result in a small preview window, scrolling through individual stories dozens of times, adjusting the subtitles, changing filters and adding music. And what if you mount a project with small video files, and already carry out the final assembly with the originals? Anyway, we probably rarely reveal the full screen preview during operation. In the end, the process would look like this:
Everything in order.
For Linux, there is a fairly popular program for converting and compressing video - HandBrake. Since the task is to simply compress the source code, I will not specifically invent with the settings and just use ready-made presets. Select the iPod, which gives a good result (recall, video for home use) and try to compress the source.
Sometimes after a great holiday, we can have dozens, if not hundreds of these sources, and we need to automate this work. Specifically, HandBrake has its own command interface . Therefore, you can quickly roll the script engine, well, something like this:
By running it with the convert.sh ./original-videos ./small-videos command , we will create a smaller and optimized copy for each file. The only condition is to make life easier for us in the future, we agree that all files and their clones will have the same name (although, most likely, different extensions). Here is an example of what I did. Two folders:
Thus, it is then possible to replace the copies with their originals by program name. After that, the folder with the original video is sent to the server, and small copies remain on our laptop for further work.
An example of the original video and its lightweight clone:
Um ... have I gone too far with squeezing? All just for the sake of the purity of the experiment!
So, well, started the project. The quality of the preview is disgusting, of course, but what to do: we wanted to get rid of the vile lags that slow down all the work. Continuing to cry and prick, we mount the film. While everything works smartly, especially without straining the processor. The critical moment: the time has come for information. We press the cherished button Generate Script ... , save the result. Now let's see what exactly happened and what the program has saved.
In the KDEnlive home folder (the default is ~ / kdenlive / scripts / ), two files appear:
The shell file my-movie_001.sh has a short set of commands:
There is no special street magic here: it just keeps the paths and settings in itself, and then starts the render with the necessary parameters. If desired, they can be edited, but it is worth paying attention to one parameter - “threads = 1”, which, as you might guess, sets the number of threads. Depending on the processor power, this number can be increased by significantly increasing the rendering performance. Moreover, we are going to run it on a more powerful computer, without loading our laptop.
But of particular interest for this experiment is still another file, which is marked with a meaningful declaration.
What kind of MLT file is this?
It turns out (at least for me), in the world of open source software, there is such a format for transmitting video broadcasts, as well as the eponymous framework called MLT . The site says that it is:
With the help of the console program melt, you can reduce video projects directly to the console. That is, in theory, you can even make a whole movie without leaving the console. Just to do this, you need to write and run a kilometer team, which most, of course, do not like. Instead, the entire project is described in a special MLT format, which is based on ubiquitous XML. Thus, video rendering is the launch of the console melt program, which is fed both to the parameters of the future movie and the project itself in the MLT format. The MLT file itself is generated by a GUI program. In my case, as I mentioned, it was KDenlive, with which, by the way, melt is supplied as a dependency. More information about using this utility in offdok .
Now let's look at the structure of the MLT file itself, generated from our movie:
Despite the volume and apparent confusion, this file has a fairly simple structure:
More information about the structure of the mlt file here .
So, we have generated the first mlt-file, which will contain the structure of the film itself, composed of compressed video clones. Every time we are ready to render a movie, we will generate this file with the structure. It seems to have figured out how the project turns out on the client side (i.e., on the laptop), now we need to figure out how to work with this on the server before rendering itself. The plan is to replace all the “producers” in the mlt-file with small videos with exactly the same ones, but with the original ones, and then run the mixing. Ideally, no one will notice the substitution.
For this operation, you need to know exactly which small files correspond to large originals. The problem is further complicated by the fact that the “producer” uses the technical data of the video clip. Here is an example of one of them:
Not bad, yes? It is necessary to somehow replace all these parameters of the resource and preferably not manually. The problem is further complicated by the fact that there is no scheme for mlt, thus it is programmatically difficult to verify the correctness of the file. We risk and try to do without the scheme.
How will we replace the meta-data of the clones with the original ones (which, of course, will be different, since the videos themselves are different)? Fortunately, the melt is smart enough to make the correct mlt, which it will consume later. When we start rendering with the melt command, we need to specify the data sources ("producer", as shown above) and the consumer, where the stream is redirected ("consumer"). The consumer is different, all options can be viewed here or by typing a command from the console:
As you can see, you can redirect rendering output to anything. For example, during normal rendering to a file, avformat is used. Everything we mount will be saved in one file. As a second example, it is possible to send the output immediately to stdout using the command:
Then the window with video playback will open and melt will flatten the movie, at the same time redirecting it to the window that opens on the fly.
But if you specify “xml” as a consumer and specify video files as the source, then melt will list all the files and their technical parameters and output all the content in XML format into a beautifully designed MLT file. Here is an example command:
It will generate a MLT file containing only “producers” for video files in this directory with MOV resolution. Naturally, with the correct meta-data of the original (large) video files. So this is exactly what is needed for our cunning plan! So:
As a result, each producer like this:
Must be replaced by a producer like this:
In order to accomplish this operation, I sketched a small Python script with a test . Actually, I'm a Java follower, so this is my first Python code. If after reading you think that you need to straighten my crooked hands, then your thoughts and suggestions are welcome in the form of a pull request.
Run this script with this command:
which says "take all the producers from the original.mlt file and replace them with the corresponding ones in the small.mlt file, but do not touch the rest."
Well, now it's time to try building a project remotely. As it should in Linux, we will create on the server side a clear and simple shell script, ...
... which does everything we need: it generates a fresh MLT from the originals, runs the Python script to replace the sources and starts the rendering. At the end, you can insert an alert via email or SMS.
Note I would like to comment on a small subtlety, which is described in Step 1 and Step 4 in the above script. As I mentioned above, in addition to videos and pictures, other mlt files can also be used as source files. , .. Clip Job . , .
, , . {fileName}.{extension}.mlt . :
, Step 1 , melt . , IMG_3764.mp4.mlt IMG_3764.MOV.mlt . Step 4 , mlt.
, .
, , mlt . . , , , , , .
, , :
, , , MLT- , , , , , .
, , -. . ,
.
! , , – !
As you know, rendering of video and three-dimensional images takes a lot of time and requires a lot of computer resources. My memories are still fresh: as a student, I put a humming computer with a running movie build program at night, and in the morning it turned out that either there was not enough space on my hard drive, or I forgot to add something to the video and I had to start all over again. Now real professionals do this operation remotely. For example, during the filming of Avatar, director James Cameron made an order for himself to create a whole data center, in which scenes were rendered. I'm still not so cool to build my own separate data center, but the idea that this operation could be run separately on another server did not give me rest for a long time.
')
The second problem is the installation of the video itself. Despite the fact that modern computers are becoming more powerful, the development of video cameras also does not stand still and the resulting video files are becoming harder and harder. And, as a result, it becomes problematic to handle them. When you mount a long film, the preview of the track with superimposed filters and transitions begins to load the processor and the image starts to lag, making the creative installation process tedious. From what I have tried, the most productive was iMovie, which is preinstalled on all Macs. Even a 45-minute film could be edited without any problems on a relatively low-power macbook. You could apply any filter and see the result in the preview window without any lags. So the poppy owners are lucky here. One minus in iMovie: during operation, it begins to consume an inordinate amount of disk space. Apparently, this is due to aggressive caching for editing and previewing.
In this article I will describe my experience as I tried to solve these two problems of editing and rendering with the example of a home video clip.
Tool
If you compare with Windows or MacOS, for Linux there are usually not so many desktop applications, but the choice of video editors turned out to be surprisingly wide. Having played with several options, I stopped at Kdenlive for working with small videos. When comparing with other programs, there is a good set of functions, but in this article we are only interested in one, but a nice feature: when mixing an already finished movie, the program offers two options - as usual, generate to a file or generate a script.
At the same time, it will simply create a regular shell script, which you can then run from the command line. Including on another computer. What am I getting at? Yes, just so you can reduce the project on a remote server. To do this, you can use the clouds, your server, or even a powerful work computer, which remains constantly turned on in the office (but I, of course, did not say that). The plan is simple: we reduce the project, generate the script, transfer the entire project to the server, start the process remotely.
But it sounds too easy for real bison. Add some conditions to the task. I have already complained that installing original video files is a task that requires a lot of resources. While working, we view the result in a small preview window, scrolling through individual stories dozens of times, adjusting the subtitles, changing filters and adding music. And what if you mount a project with small video files, and already carry out the final assembly with the originals? Anyway, we probably rarely reveal the full screen preview during operation. In the end, the process would look like this:
- For each video source, we will create its lightweight clone.
- We will transfer originals to the server.
- Installation will be done on a personal computer using lightweight copies (I used my laptop).
- Before the assembly, we transfer the project to the server, where the source code is already in its original quality.
- Replace lightweight clones with original ones.
- We launch the rendering of the film in high resolution.
- Optional: notify on completion of the process by email or SMS.
- Download the finished result on your personal computer.
- We are dragged from the finished video (and own skill).
Everything in order.
Lightweight copies
For Linux, there is a fairly popular program for converting and compressing video - HandBrake. Since the task is to simply compress the source code, I will not specifically invent with the settings and just use ready-made presets. Select the iPod, which gives a good result (recall, video for home use) and try to compress the source.
Sometimes after a great holiday, we can have dozens, if not hundreds of these sources, and we need to automate this work. Specifically, HandBrake has its own command interface . Therefore, you can quickly roll the script engine, well, something like this:
#!/bin/bash # convert.sh quality=30 for file in ls $1/* do tempfile="${file##*/}" filename="${tempfile%.*}" outputFile="$2/$filename.mp4" echo -e "\n\n Compress file $file" HandBrakeCLI -i $file -o $outputFile --preset="iPod" -q $quality -w 160 done By running it with the convert.sh ./original-videos ./small-videos command , we will create a smaller and optimized copy for each file. The only condition is to make life easier for us in the future, we agree that all files and their clones will have the same name (although, most likely, different extensions). Here is an example of what I did. Two folders:
/ small-videos / original-videos FILE0001.mp4 FILE0001.MOV FILE0002.mp4 FILE0002.MOV FILE0003.mp4 FILE0003.MOV ... ...
Thus, it is then possible to replace the copies with their originals by program name. After that, the folder with the original video is sent to the server, and small copies remain on our laptop for further work.
An example of the original video and its lightweight clone:
Um ... have I gone too far with squeezing? All just for the sake of the purity of the experiment!
Project
So, well, started the project. The quality of the preview is disgusting, of course, but what to do: we wanted to get rid of the vile lags that slow down all the work. Continuing to cry and prick, we mount the film. While everything works smartly, especially without straining the processor. The critical moment: the time has come for information. We press the cherished button Generate Script ... , save the result. Now let's see what exactly happened and what the program has saved.
In the KDEnlive home folder (the default is ~ / kdenlive / scripts / ), two files appear:
- shell script my-movie_001.sh
- and my-movie_001.sh.mlt
The shell file my-movie_001.sh has a short set of commands:
#! /bin/sh SOURCE="/home/w32blaster/kdenlive/scripts/my-movie_001.sh.mlt" TARGET="/home/w32blaster/kdenlive/my-movie.mp4" RENDERER="/usr/bin/kdenlive_render" MELT="/usr/bin/melt" PARAMETERS="-pid:8613 $MELT hdv_1080_50i avformat - $SOURCE $TARGET properties=x264-medium g=120 crf=20 ab=160k threads=1 real_time=-1" $RENDERER $PARAMETERS There is no special street magic here: it just keeps the paths and settings in itself, and then starts the render with the necessary parameters. If desired, they can be edited, but it is worth paying attention to one parameter - “threads = 1”, which, as you might guess, sets the number of threads. Depending on the processor power, this number can be increased by significantly increasing the rendering performance. Moreover, we are going to run it on a more powerful computer, without loading our laptop.
But of particular interest for this experiment is still another file, which is marked with a meaningful declaration.
SOURCE = "/ home / w32blaster / kdenlive / scripts / my-movie_001.sh.mlt"
What kind of MLT file is this?
MLT and project description
It turns out (at least for me), in the world of open source software, there is such a format for transmitting video broadcasts, as well as the eponymous framework called MLT . The site says that it is:
MLT is an open source multimedia framework, designed and developed for television broadcasting. It provides a toolkit for broadcasters, video editors, media players, transcoders, web streamers. The functionality of the plug-in based API.
With the help of the console program melt, you can reduce video projects directly to the console. That is, in theory, you can even make a whole movie without leaving the console. Just to do this, you need to write and run a kilometer team, which most, of course, do not like. Instead, the entire project is described in a special MLT format, which is based on ubiquitous XML. Thus, video rendering is the launch of the console melt program, which is fed both to the parameters of the future movie and the project itself in the MLT format. The MLT file itself is generated by a GUI program. In my case, as I mentioned, it was KDenlive, with which, by the way, melt is supplied as a dependency. More information about using this utility in offdok .
Now let's look at the structure of the MLT file itself, generated from our movie:
<?xml version='1.0' encoding='utf-8'?> <mlt title="Anonymous Submission" version="0.9.3" root="/home/w32blaster" LC_NUMERIC="en_GB.UTF-8"> <producer in="0" out="5367" id="117_4"> <property name="resource">Videos/small/FILE0001.mp4</property> ... - </producer> <playlist id="playlist1"> <entry in="0" out="2033" producer="117_4"> <filter out="1838" id="volume"> ... ... </filter> </playlist> <tractor title="Anonymous Submission" global_feed="1" in="0" out="17018" id="maintractor"> <property name="meta.volume">1</property> <track producer="black_track"/> <track hide="video" producer="playlist1"/> <track hide="video" producer="playlist2"/> <track producer="playlist3"/> <track producer="playlist4"/> <track producer="playlist5"/> <transition id="transition0"> <property name="a_track">1</property> <property name="b_track">2</property> <property name="mlt_type">transition</property> <property name="mlt_service">mix</property> <property name="always_active">1</property> <property name="combine">1</property> <property name="internal_added">237</property> </transition> .... - , .... </tractor> </mlt> Despite the volume and apparent confusion, this file has a fairly simple structure:
- mlt - the main tag, which declares that it is nothing more than a description of the video in mlt-format. Also, the first project parameters are declared, which are removed from the example for simplicity.
- producer is the object that is the source of the image. It can be a video, music, an image, just a color fill, captions, and even another mlt file (more on that below). Inside there are many parameters, and in the case of working with files, the path to the resource is important. For each resource that is added to our project, a separate “producer” is generated.
- playlist - a playlist that consists of all the "producers". Filters can also be here (in the example above, this is a sound attenuation filter). For each resource we specify which segment should be inserted. That is, when we cut the edges of the clip on the timeline, then at the MLT format level we simply set the in and out parameters of the entry tag. Well, the entry tag itself has a link to the “producer”.
- tractor - in fact, this is our entire film, or timeline, as it is also called. What is officially called non-linear editing, when we change the order and drag video clips among themselves on the tracker - everything is described here. All transitions, overlays, captions and everything-everything-everything is described by the “tractor” tag.
More information about the structure of the mlt file here .
So, we have generated the first mlt-file, which will contain the structure of the film itself, composed of compressed video clones. Every time we are ready to render a movie, we will generate this file with the structure. It seems to have figured out how the project turns out on the client side (i.e., on the laptop), now we need to figure out how to work with this on the server before rendering itself. The plan is to replace all the “producers” in the mlt-file with small videos with exactly the same ones, but with the original ones, and then run the mixing. Ideally, no one will notice the substitution.
For this operation, you need to know exactly which small files correspond to large originals. The problem is further complicated by the fact that the “producer” uses the technical data of the video clip. Here is an example of one of them:
<producer in="0" out="1327" id="65_2"> <property name="mlt_type">producer</property> <property name="length">1328</property> <property name="eof">pause</property> <property name="resource">Videos/Austria2015-small/FILE1235.mp4</property> <property name="meta.media.nb_streams">2</property> <property name="meta.media.0.stream.type">video</property> <property name="meta.media.0.stream.frame_rate">30</property> <property name="meta.media.0.stream.sample_aspect_ratio">0</property> <property name="meta.media.0.codec.width">160</property> <property name="meta.media.0.codec.height">90</property> <property name="meta.media.0.codec.frame_rate">90000</property> <property name="meta.media.0.codec.pix_fmt">yuv420p</property> <property name="meta.media.0.codec.sample_aspect_ratio">0</property> <property name="meta.media.0.codec.colorspace">709</property> <property name="meta.media.0.codec.color_trc">1</property> <property name="meta.media.0.codec.name">h264</property> <property name="meta.media.0.codec.long_name">H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10</property> <property name="meta.media.0.codec.bit_rate">52987</property> <property name="meta.media.1.stream.type">audio</property> <property name="meta.media.1.codec.sample_fmt">fltp</property> <property name="meta.media.1.codec.sample_rate">48000</property> <property name="meta.media.1.codec.channels">2</property> <property name="meta.media.1.codec.name">aac</property> <property name="meta.media.1.codec.long_name">AAC (Advanced Audio Coding)</property> <property name="meta.media.1.codec.bit_rate">163117</property> <property name="meta.attr.1.stream.language.markup">eng</property> <property name="meta.attr.major_brand.markup">mp42</property> <property name="meta.attr.minor_version.markup">512</property> <property name="meta.attr.compatible_brands.markup">isomiso2avc1</property> <property name="meta.attr.encoder.markup">HandBrake 0.10.1 2015030800</property> <property name="seekable">1</property> <property name="meta.media.sample_aspect_num">1</property> <property name="meta.media.sample_aspect_den">1</property> <property name="aspect_ratio">1</property> <property name="audio_index">1</property> <property name="video_index">0</property> <property name="mlt_service">avformat</property> <property name="meta.media.frame_rate_num">30</property> <property name="meta.media.frame_rate_den">1</property> <property name="meta.media.colorspace">709</property> <property name="meta.media.width">160</property> <property name="meta.media.height">90</property> <property name="meta.media.top_field_first">0</property> <property name="meta.media.progressive">1</property> <property name="meta.media.color_trc">1</property> </producer> Not bad, yes? It is necessary to somehow replace all these parameters of the resource and preferably not manually. The problem is further complicated by the fact that there is no scheme for mlt, thus it is programmatically difficult to verify the correctness of the file. We risk and try to do without the scheme.
How will we replace the meta-data of the clones with the original ones (which, of course, will be different, since the videos themselves are different)? Fortunately, the melt is smart enough to make the correct mlt, which it will consume later. When we start rendering with the melt command, we need to specify the data sources ("producer", as shown above) and the consumer, where the stream is redirected ("consumer"). The consumer is different, all options can be viewed here or by typing a command from the console:
$ melt -query "consumers" --- consumers: - decklink - xgl - qglsl - sdl - sdl_audio - sdl_preview - sdl_still - cbrts - xml - sdi - jack - avformat - multi - null - gtk2_preview - blipflash - rtaudio ... As you can see, you can redirect rendering output to anything. For example, during normal rendering to a file, avformat is used. Everything we mount will be saved in one file. As a second example, it is possible to send the output immediately to stdout using the command:
$ melt original.mlt -consumer libdv
Then the window with video playback will open and melt will flatten the movie, at the same time redirecting it to the window that opens on the fly.
But if you specify “xml” as a consumer and specify video files as the source, then melt will list all the files and their technical parameters and output all the content in XML format into a beautifully designed MLT file. Here is an example command:
$ melt * .MOV -consumer xml: original.mlt
It will generate a MLT file containing only “producers” for video files in this directory with MOV resolution. Naturally, with the correct meta-data of the original (large) video files. So this is exactly what is needed for our cunning plan! So:
- We know where the originals are on the server.
- Based on these originals, we generate in the command line a MLT file that contains only one “producers” with original videos and their meta data. Let's call it original.mlt . In this file there is no description of the film (that is, the tractor tag), only the source code. We will use it as a source of so-called. the right "producers."
- We are transferring the current MLT file from the laptop to the server, which contains our film with the current changes, but with “spoiled” small copies. Let's call it small.mlt
- Keeping in mind that the originals and copies have the same name (but, possibly, different extensions), we can match the “correct” producers from the first MLT file with the “cut” ones from the second MLT file
- It is necessary to replace not only the producers themselves, but also their IDs, since they will most certainly be different.
- You can start to note.
As a result, each producer like this:
<producer> <property name="resource">Videos/small/FILE1234.mp4</property> ... - </producer> Must be replaced by a producer like this:
<producer> <property name="resource">Videos/original/FILE1234.MOV</property> ... - </producer> In order to accomplish this operation, I sketched a small Python script with a test . Actually, I'm a Java follower, so this is my first Python code. If after reading you think that you need to straighten my crooked hands, then your thoughts and suggestions are welcome in the form of a pull request.
Listing 1: MLT file handler code
#!/usr/bin/env python import xml.etree.ElementTree as ET import os.path import sys, getopt def main(argv): ''' The main purpose of this programm is to take the given MLT file containing producers with an compressed ("small") resources (video, images, sound) and replace them with identical producers with uncompressed ("original") resources from another MLT file Example of usage: python process.py -s ~/Videos/small.sh.mlt -p ~/Videos/original/original.mlt ''' smallVideosFile, originalVideosFile, homeDir = _extractCLArguments(argv) if (smallVideosFile == '' or originalVideosFile == ''): print "Error. Both arguments -s and -p must be provided!" sys.exit(2) print "Start MLT file processor. We will modify %s using data from %s" % (smallVideosFile, originalVideosFile) # Parse the MLT file with small resources smallTree = ET.parse(smallVideosFile) smallTreeRoot = smallTree.getroot() # and MLT file with original (big, full size) resources originalTree = ET.parse(originalVideosFile) originalTreeRoot = originalTree.getroot() # prepare maps "video file name" <==> "producer ID" mapSmallProducers = _getMapOfProducerIds(smallTreeRoot) mapOriginalProducers = _getMapOfProducerIds(originalTreeRoot) # build map "small producer ID" <==> "original producer ID", having the same video resource mapID = _mergeMaps(mapSmallProducers, mapOriginalProducers) # update 'root' value in MLT tag _updateRootTag(homeDir, smallTreeRoot) # then replace all the producers containing small videos with those with original videos _replaceAllSmallProducersWithOriginal(mapOriginalProducers, smallTreeRoot, originalTreeRoot) # print modified MLT to output file ET.ElementTree(smallTreeRoot).write(smallVideosFile, encoding='utf-8', xml_declaration=True) def _extractResourceName(fullName): ''' Extracts only the resource name from full path. For example, if the fullName is Videos/small/FILE1114.mkv this method returns FILE1114 ''' if "?" in fullName: withoutParam = fullName.split("?",1)[0] return os.path.splitext(os.path.basename(withoutParam))[0] else: return os.path.splitext(os.path.basename(fullName))[0] def _mergeMaps(mapSmallProducers, mapOriginalProducers): ''' Builds map of producer IDs: each small producer's ID should match appropriate big producer's ID, having the same resource. ''' mapID = {} for fileName, oldProducerId in mapSmallProducers.items(): if (fileName in mapOriginalProducers): mapID[oldProducerId] = mapOriginalProducers[fileName] return mapID def _getMapOfProducerIds(tree): ''' Builds the map for the given file: "resource (video file name)" <=> "producer ID". ''' arrProducers = tree.findall('producer') # Collect the map "resourse (video file name)" <==> "id of the producer" mapIdToResource = {} for producer in arrProducers: resourceType = producer.find("*[@name='mlt_service']").text isReplacementNeeded = (resourceType == "avformat" or resourceType == "framebuffer" or resourceType == "xml") if (isReplacementNeeded): resourceValue = producer.find("*[@name='resource']").text # extract only filename (without path and extension) fileName = _extractResourceName(resourceValue) mapIdToResource[fileName] = producer.attrib.get('id') else: print "[Ignored] the producer %s (resource type is %s) is ignored" % (producer.attrib.get('id'), resourceType) pass return mapIdToResource def _replaceAllSmallProducersWithOriginal(mapOriginalProducers, rootToBeModified, rootOriginal): ''' Replaces all DOM-element 'Producer' containing small resources with identical producers, containing the original resource ''' print "Replace old producer having small size with another one having original resource:" arrSmallProducers = rootToBeModified.findall('producer') for producer in arrSmallProducers: resourceType = producer.find("*[@name='mlt_service']").text if (resourceType == "avformat"): _replaceProducerAvformat(producer, mapOriginalProducers, rootToBeModified, rootOriginal) elif (resourceType == "framebuffer"): _replaceProducerFramebuffer(producer, mapOriginalProducers, rootToBeModified, rootOriginal) elif (resourceType == "xml"): _replaceProducerXml(producer, mapOriginalProducers, rootToBeModified, rootOriginal) def _replaceProducerAvformat(producer, mapOriginalProducers, rootToBeModified, rootOriginal): ''' Replace entirely one producer, containing small resource with similar producer from original tree. Resulting producer should have the same ID, but it must get all the values from original tree (URLs, codec properties and other technical information) ''' resourceValue = producer.find("*[@name='resource']").text fileName = _extractResourceName(resourceValue) if(fileName in mapOriginalProducers): # remember in, out and length from small producer attrIn = producer.attrib.get('in') attrOut = producer.attrib.get('out') length = producer.find("*[@name='length']").text producerId = producer.attrib.get('id') originalId = mapOriginalProducers[fileName] origProducer = rootOriginal.find('.//producer[@id="' + originalId + '"]') if origProducer is not None: rootToBeModified.remove(producer) # do not touch original node, deal with clone instead origProducerClone = origProducer.copy() # update in, out and length to match with small producer origProducerClone.set('in', attrIn) origProducerClone.set('out', attrOut) origProducerClone.set('id', producerId) origProducerClone.find("*[@name='length']").text = length rootToBeModified.insert(1, origProducerClone) print " - Avformat resource (id=%s) is replaced with resource (id=%s)" % (producerId, originalId) else: print " - [Ignored] Avformat resource (id=%s) was not found. File name is '%s' and found original ID is '%s'" % (producerId, fileName, originalId) def _replaceProducerFramebuffer(producer, mapOriginalProducers, rootToBeModified, rootOriginal): ''' Update given producer's resource. The resulting producer should have the same properties except resource path. Keep in mind, that we need to keep the parameter after "?" sign. ''' resourceNode = producer.find("*[@name='resource']") resourceValue = resourceNode.text producerId = producer.attrib.get('id') fileName = _extractResourceName(resourceValue) if(fileName in mapOriginalProducers): originalId = mapOriginalProducers[fileName] origProducer = rootOriginal.find('.//producer[@id="' + originalId + '"]') if origProducer is not None: originalResourcePath = origProducer.find("*[@name='resource']").text extension = resourceValue.split("?",1)[1] resourceNode.text = originalResourcePath + "?" + extension print " - Framebuffer resource (id=%s) is replaced with resource (id=%s)" % (producerId, originalId) else: print " - [Ignored] the Framebuffer resource (id=%s) is ignored!" % (producerId) def _replaceProducerXml(producer, mapOriginalProducers, rootToBeModified, rootOriginal): ''' Update given producer in case if it is XML resource. The resulting producer should have updated resource path including .mlt extension. ''' resourceNode = producer.find("*[@name='resource']") resourceValue = resourceNode.text[:-4] # trim .mlt extension producerId = producer.attrib.get('id') fileName = _extractResourceName(resourceValue) if(fileName in mapOriginalProducers): originalId = mapOriginalProducers[fileName] origProducer = rootOriginal.find('.//producer[@id="' + originalId + '"]') if origProducer is not None: originalResourcePath = origProducer.find("*[@name='resource']").text resourceNode.text = originalResourcePath + ".mlt" print " - XML resource (id=%s) is replaced with resource (id=%s)" % (producerId, originalId) def _updateRootTag(homedir, rootToBeModified): ''' In order to find all the resources, MLT uses the arggument "root" placed in tag MLT. When we work on different computers, we must also update this value in order MELT would be able to find resources. ''' if (homedir != ''): rootToBeModified.set('root', homedir) def _extractCLArguments(argv): inputfile = '' outputfile = '' homedir = '' try: opts, args = getopt.getopt(argv,"hs:p:u:",["smallfile=","producersfile=", "userhomedir="]) except getopt.GetoptError: print 'process.py -s <smallFileToBeModified> -p <producersFile>' sys.exit(2) for opt, arg in opts: if opt == '-h': print '''process.py -s <smallFile> -p <producersFile> -u <userHomeDir> where "smallFile" is a mlt file to be modified and "producersFile" is a mlt file containing the list of producers with original resources. ''' sys.exit() elif opt in ("-s", "--smallfile"): inputfile = arg elif opt in ("-p", "--producersfile"): outputfile = arg elif opt in ("-u", "--userhomedir"): homedir = arg return inputfile, outputfile, homedir if __name__ == "__main__": main(sys.argv[1:]) Run this script with this command:
# python process.py -s small.mlt -p original.mlt
which says "take all the producers from the original.mlt file and replace them with the corresponding ones in the small.mlt file, but do not touch the rest."
Putting it all together
Well, now it's time to try building a project remotely. As it should in Linux, we will create on the server side a clear and simple shell script, ...
Listing 2: full server side script
#!/bin/bash SMALL_RESORCES="Small" ORIGINAL_RESOURCES="Original" KDENLIVE_SCRIPT="austria-2015_001.sh" PY_SCRIPT="/home/ilya/home-workspace/mlt-producer-replacer/process.py" CURRENT_DIR="/home/ilya/Videos/MltProcessor" rm log echo -e "\n\n Step 1. Rename all the .mp4.mlt resources to .MOV.mlt" rm $ORIGINAL_RESOURCES/*.MOV.mlt for file in $ORIGINAL_RESOURCES/*.mlt; do cp "$file" "$ORIGINAL_RESOURCES/`basename $file .mp4.mlt`.MOV.mlt" done echo -e "\n\n Step 2. Generate fresh list of producers with original sources" rm $CURRENT_DIR/original.mlt cd ~ melt $CURRENT_DIR/$ORIGINAL_RESOURCES/*.{MOV,mp3} -consumer xml:original.mlt cd $CURRENT_DIR mv ~/original.mlt $CURRENT_DIR echo -e "\n\n Step 3. Update MLT and replace small videos with original ones" # process the main MLT file cp $KDENLIVE_SCRIPT.mlt $KDENLIVE_SCRIPT.mlt-BACKUP python $PY_SCRIPT -s $KDENLIVE_SCRIPT.mlt -p $CURRENT_DIR/original.mlt -u $HOME echo -e "\n\n Step 4. Update additional MLT files" # process other MLT resources, that are producers with type=xml for i in $ORIGINAL_RESOURCES/*.MOV.mlt; do python $PY_SCRIPT -s $i -p $CURRENT_DIR/original.mlt -u $HOME done echo -e "\n\n Step 5. Run rendering" ./$KDENLIVE_SCRIPT echo -e "\n\n Step 6. Make it smaller" HandBrakeCLI -i austria.mp4 -o austria.small.m4v --preset="Universal" echo "FINISHED" >> log ... which does everything we need: it generates a fresh MLT from the originals, runs the Python script to replace the sources and starts the rendering. At the end, you can insert an alert via email or SMS.
Note I would like to comment on a small subtlety, which is described in Step 1 and Step 4 in the above script. As I mentioned above, in addition to videos and pictures, other mlt files can also be used as source files. , .. Clip Job . , .
, , . {fileName}.{extension}.mlt . :
<producer id="12_1" in="0" out="375"> <property name="mlt_type">producer</property> <property name="resource">Videos/MltProcessor/Original/IMG_3764.MOV.mlt</property> <property name="mlt_service">xml</property> .... </producer> , Step 1 , melt . , IMG_3764.mp4.mlt IMG_3764.MOV.mlt . Step 4 , mlt.
, .
- ( ), .
- GUI Kdenlive, .
- , , Generate file, shell-. , threads . .
- , shell- .
- . , mlt.
- shell- ( 2).
, , mlt . . , , , , , .
, , :
#!/bin/bash SERVER_HOSTNAME="10.20.30.40" USERNAME="ilya" REMOTE_PROJECT_DIR="/home/ilya/Videos/MltProcessor" REMOTE_ORIGINAL_DIR="/home/ilya/Videos/MltProcessor/Original" LOCAL_MLT="/home/w32blaster/kdenlive/scripts/austria-2015_001.sh.mlt" LOCAL_SOURCES_DIR="/home/w32blaster/Videos/Austria2015-small" # 1. Upload main MLT file scp $LOCAL_MLT $USERNAME@$SERVER_HOSTNAME:$REMOTE_PROJECT_DIR # 2. Upload all others MLT files, that represents producer sources scp $LOCAL_SOURCES_DIR/*.mlt $USERNAME@$SERVER_HOSTNAME:$REMOTE_ORIGINAL_DIR # 3. Execute rendering on the remote server ssh $USERNAME@$SERVER_HOSTNAME "$REMOVE_PROJECT_DIR/run.sh" , , , MLT- , , , , , .
Conclusion
, , -. . ,
.
- , Kdenlive,
- , . : .
- , , , , , - , :)
! , , – !
Source: https://habr.com/ru/post/260837/
All Articles