Paralleling tasks in Linux
I needed to recode a number of video files. For this, I wrote the following script:
It seemed, everything would be ready, but I noticed that only one of the two processors was loaded, which means that this process can be accelerated twice a time.
')
You can set mencoder options. For x264 codec it is possible to specify the number of streams:
The method is good, but not universal and the quality of the result may deteriorate.
The method is simple, we start processes in parallel, for example:
The disadvantage of the method is that due to the fact that files of different sizes, the time for their processing can vary significantly and as a result, one processor can be unloaded much earlier than the other.
I wanted to somehow improve the scenario, so that two tasks would always work in parallel, and as they were completed, new ones were launched. Reflecting on how I would do this, I remembered that there is a wonderful build utility that can do exactly what I need. It turned out the following:
Makefile:
It turned out surprisingly simple and short. The list of all the files I want to get is listed first. It is followed by the path to the source files and the build rule. Run with the command “make -j 2” so that 2 processes work simultaneously.
(The first article on Habré, do not judge strictly)
#!/bin/bash
recode() {
mencoder -o $2 $1 -ovc x264 -x264encopts bitrate=22000:keyint=50 -oac mp3lame -lameopts vbr=3:br=320 -fps 50
}
recode input/00108.mts 00108.avi
recode input/00109.mts 00109.avi
...
...
It seemed, everything would be ready, but I noticed that only one of the two processors was loaded, which means that this process can be accelerated twice a time.
The first method: paralleling by means of mencoder
')
You can set mencoder options. For x264 codec it is possible to specify the number of streams:
- threads = <0-16>
Use streams to encode simultaneously on multiple processors (default: 1). This slightly degrades the coding quality. 0 or 'auto' - automatically determine the number of processors and use the appropriate number of threads.
The method is good, but not universal and the quality of the result may deteriorate.
Second way: parallelizing with Bash
The method is simple, we start processes in parallel, for example:
(recode input/00108.mts 00108.avi
recode input/00109.mts 00109.avi
...
...) &
(recode input/00108.mts 00110.avi
recode input/00109.mts 00111.avi
...
...)
The disadvantage of the method is that due to the fact that files of different sizes, the time for their processing can vary significantly and as a result, one processor can be unloaded much earlier than the other.
Third way: parallelizing with GNU make
I wanted to somehow improve the scenario, so that two tasks would always work in parallel, and as they were completed, new ones were launched. Reflecting on how I would do this, I remembered that there is a wonderful build utility that can do exactly what I need. It turned out the following:
Makefile:
all: 00108.avi 00109.avi 00110.avi 00111.avi 00118.avi 00119.avi 00120.avi 00123.avi
VPATH = input:.
%.avi: %.mts
mencoder -o $@ $< -ovc x264 -x264encopts bitrate=22000:keyint=50 -oac mp3lame -lameopts vbr=3:br=320 -fps 50
It turned out surprisingly simple and short. The list of all the files I want to get is listed first. It is followed by the path to the source files and the build rule. Run with the command “make -j 2” so that 2 processes work simultaneously.
(The first article on Habré, do not judge strictly)
Source: https://habr.com/ru/post/66903/
All Articles