Matlab is a well-known numerical computing environment widely used by the scientific community. Works in Windows, nix-systems and on Macs. The Matlab language is optimized for working with matrices and multidimensional arrays; in addition, there are a huge number of extensions (officially called Toolboxes) for solving optimization problems, statistical calculations, signal and image processing, etc. Plus, there is a large amount of scientific code already written on Matlab, which additionally spurs its popularity.

In this article I will briefly describe the possibilities of Matlab to work with video. For those who are not familiar with the matlab syntax, a brief overview of the main features in Russian can be read here .

Video format

The first thing you need to do to work with video is to upload it to Matlab. The main difficulty that arose in me is related to codecs. By default, the list of supported video formats is small ( English documentation ) and whether a matlab will open your video depends on the codecs installed in the system. The general rule is that if your built-in player (Windows Media Player in Windows) cannot open a file, then Matlab most likely will not open it.
')
Accordingly, the ability to record video is highly dependent on the installed codecs, although Matlab will be able to create uncompressed video on any system without installed codecs.

Problems with reading and recording video, I decided to install the KLite Codec Pack Full in the version of "Lots of stuff":

Different ways to read videos

There are several ways to read videos in the lab. About two of them are described below.

Reading video using mmreader and read methods

The first way to read video is to use the mmreader method to open a file and further use the read method to read frames:

 % open file
 video = mmreader ('d: \ test \ 1.avi');

 % like this you can learn about some properties of the open video
 width = video.Width;
 height = video.Height;
 frameRate = video.FrameRate;
 numOfFrames = video.NumberOfFrames;

 % read frame
 % frames numbered from 1
 frameNo = 50;
 frame = read (video, frameNo);

 % frame can be read and like this
 frame = video.read (frameNo);

 % read frame range
 % 50 - the number of the first frame read
 % 100 - last read frame number
 framesRange = [50 100];
 frames = read (video, framesRange);

 % read all videos
 % operation will not be performed if the video is large and does not fit into memory
 entireVideo = read (video);

 % There is no special function to close the video file.
 % Correspondingly, when you finish working with video
 % no additional action is needed.

All frame reading functions return a multidimensional matrix as the execution result. For example, here is the result of executing several commands for working with video in the interactive desktop lab:

  >> video.Width

 ans =

    704

 >> video.Height

 ans =

    576

 >> frame = read (video, 5);
 >> size (frame)

 ans =

    576,704 3

 >> frames = read (video, [5 10]);
 >> size (frames)

 ans =

    576 704 3 6 

Thus, as a result of reading one frame, a matrix is ​​obtained with the size of Height x Width x 3, the elements of which are 8-bit unsigned integers. As a result of reading a range of several frames, a matrix is ​​obtained with the size of Height x Width x 3 x NumOfFrames.

Correspondingly, when you have loaded a frame, you can work with it as with an ordinary matlab matrix, and also apply various functions for image processing included in the Image Processing Toolbox to it.

Reading video with System Objects - Matlab 2010a innovations

The concept of system objects was introduced in the hardware version 2010a. The Matlab manual says that system objects provide an object-oriented implementation of algorithms. Compared to ordinary Matlab functions, they automatically control their state and are well suited for stream processing. In general, the main point is that according to the developers, it is more convenient to use them for stream processing of information. At the expense of convenience, you can argue, but the code becomes a bit shorter.

Here is an example of frame-by-frame video reading using system objects:

 % create an object to read the video
 reader = video.MultimediaFileReader ('Filename', 'd: \ test \ 1.avi');

 % read all videos frame by frame
 % step function will return a matrix with the size of Height x Width x 3
 % true the elements of this matrix will be of the single type - 4 byte floating point numbers
 while ~ isDone (reader)
    frame = step (reader);
 end

 % here as in the previous case, you do not need to perform any closing operations
 % at the end of reading the video

A significant disadvantage of this approach is that there is no random access to the video - you can only read frames sequentially. Also, in terms of subjective sensations, opening video and reading frames using system objects takes a little longer than using mmreader and read .

Playing video

Besides the fact that the video can be read and processed, it can also be played with built-in tools.

Playing with the movie function

The movie function allows you to play a video. The feature of using this function is that you need to prepare in advance the entire video that should be played. In addition, the video should be stored in memory in a format different from that obtained by reading the video. The following is an example of how this feature can be used to play the first 200 frames of video:

 %% reading video
 filename = 'd: \ test \ 1.avi';
 video = mmreader (filename);
 nFrames = 200;

 % prepare the structure in which the information will be read
 mov (1: nFrames) = struct ('cdata', [], 'colormap', []);

 % Read frame by frame
 for k = 1: nFrames
     mov (k) .cdata = read (video, k);
 end

 %% play video
 hf = figure;
 set (hf, 'position', [150 150 video.Width video.Height])
 movie (hf, mov, 1, video.FrameRate);

In this code, a movie is prepared in the mov variable. This variable stores an array of structures. Each structure has two fields: cdata and colormap . The cdata field stores the bitmap of the frame, and the colormap field needs to store a color palette for indexed video. In this case, this is a full-fledged RGB video and the colormap empty.

Playing with system objects

Compared with the previous example, the use of system objects can significantly reduce the code:

 % create an object to read the video
 reader = video.MultimediaFileReader ('Filename', 'd: \ test \ 1.avi');

 % create an object to play the video
 player = video.VideoPlayer;

 % read frame by frame and play the video
 while ~ isDone (reader)
    frame = step (reader);
    step (player, frame);
 end 

Video recording

You can also record video in several ways. Here I will analyze 3 methods: frame by frame recording, video recording in one piece and using system objects.

I will note in advance that if you are recording a video in more than one piece, you must close the video file at the end of the operation. If the file is not closed, then until you exit the lab you cannot access, delete, or overwrite it.

Record video frame by frame

 %% prepare variables
 filename = 'd: \ test \ 1.avi';
 result_file = 'd: \ test \ result.avi';
 nFrames = 200;
 video = mmreader (filename);
 % open file for writing
 % compressor set - XVID
 writer = avifile (result_file, 'compression', 'XVID');

 %% in cycle read-modify-write
 % structuring element for dilatation
 se = strel ('rectangle', [3 3]);
 for k = 1: nFrames
     % read frame
     frame = read (video, k);
     % do frame dilatation
     frame = imdilate (frame, se);
     % write frame
     writer = addframe (writer, frame);
 end
 % sure to close the writer
 writer = close (writer); 

Here I will note the way in which the arguments of the avifile function are avifile . The first argument is the name of the file, then the arguments go in pairs — first the name of the parameter, then the value. In the above example, the avifile function avifile in addition to the file name, also passes the compression parameter with the value XVID.

As the compression parameter, you can specify one of the identifiers: 'None', 'MSVC', 'RLE', 'Cinepak', 'Indeo3' or 'Indeo5'. On * nix systems, only the option 'None' is available. In addition to the above options, you can also specify the FourCC of the desired compressor (as in the example). Here everything depends on the codecs installed in the system for encoding the video. It should be noted that not all of the installed codecs are suitable for both decoding and video encoding. Only XVID earned on my system of several tested options.

Video recording in one piece

In addition to frame-by-frame recording, you can also set up an array of frames and write them all at once to a file. An array of frames should be prepared in the same way as in the example of using the movie function to play videos. More to the point: the array should consist of structures, one of the members of the structure is the cdata field that stores the bitmap of the image, and the second is the colormap field that stores the palette for the indexed color image:

 %% reading video
 filename = 'd: \ test \ 1.avi';
 video = mmreader (filename);
 nFrames = 200;

 % prepare the structure in which the information will be read
 mov (1: nFrames) = struct ('cdata', [], 'colormap', []);

 % Read and modify the first 200 frames of video
 se = strel ('rectangle', [3 3]);
 for k = 1: nFrames
     frame = read (video, k)
     mov (k) .cdata = imdilate (frame, se);
 end

 %% video recording
 movie2avi (mov, 'd: \ test \ result.avi', 'compression', 'XVID', 'fps', 25);

The compression parameter of the movie2avi function is the same as in the previous section.

Record video using system objects

The following code demonstrates the use of system objects for recording video:

 reader = video.MultimediaFileReader ('Filename', 'd: \ test \ 1.avi');
 % open the video to record
 % file name = d: \ test \ result.avi
 % audio - no, there is video
 % video codec = MJPEG
 writer = video.MultimediaFileWriter ('Filename', 'd: \ test \ result.avi', ...
     'AudioInputPort', false, 'VideoInputPort', true, ...
     'VideoCompressor', 'MJPEG Compressor');
 % structuring element for dilatation
 se = strel ('rectangle', [3 3]);
 while ~ isDone (reader)
    % read frame
    frame = step (reader);
    % do frame dilatation
    frame = imdilate (frame, se);
    % write frame
    step (writer, frame);
 end

 % sure to close the writer
 close (writer);

The arguments of the video.MultimediaFileWriter function are the pairs 'parameter name', the value of the parameter. Even the file name is transferred in this way (i.e., the pair 'Filename', 'file name' is indicated).

The possible values ​​of the VideoCompressor parameter here are already different from those that can be specified for the avifile and movie2avi . “None (uncompressed)” can be specified as VideoCompressor, “MJPEG Compressor” has also worked for me. If you omit the VideoCompressor parameter in the argument list, it will be equivalent to specifying 'None (uncompressed)'.

A free alternative in the face of GNU Octave

In general, the Matlab product is quite expensive, but there are some free alternatives that are partially compatible with it. One of them is GNU Octave .

GNU Octave is being actively developed and the syntax of the actual language is almost identical to Matlabovsky. Differences in the syntax are mainly due to the fact that some feature has already appeared in the trademark, and in Octave has not yet been added.

With a set of libraries, things are worse. There are no toolbox matlab in GNU Octave, although for some of the functions there are still substitutions.

Installing GNU Octave in Ubuntu 10.04

I worked with Octave in Ubuntu, so for those who want to repeat my experience I describe the installation instructions:

1. With the help of the package manager, you need to install the octave3.2 Linux packages; octave3.2-doc; octave3.2-headers with all dependencies.
2. Octave, in contrast to the matlab, consists only of an interpreter. Accordingly, you may need a GUI development environment. I installed QtOctave in Ubuntu 10.04 (also through the package manager).
3. If you plan to work with video, then you must install the Linux packages libavformat-dev, libavcodec-dev, libavutil-dev, libswscale-dev
4. In Octave itself there is also the concept of a package. Octave packages include a variety of features that extend Octave functionality. Download Octave packages from the Octave-Forge site.
To install the Octave package, you need to run Octave with administrative rights: sudo octave . And then in Octave run the command pkg install /path/to/package.tar.gz
Octave-packages image and video for working with images and video

Actually working with video in Octave

In general, loading and saving video in Octave is done in a similar way as it is done in the lab. The following is the code that dilates the first 25 frames of video:

 filename = '/home/alexey/octave/1.avi';
 result_file = '/home/alexey/octave/result.avi';

 % uncomment the next line to access the video properties
 % info = aviinfo (filename);
 % width = info.Width
 % height = info.Height

 % create a video recording object
 writer = avifile (result_file);
 se = ones ([3 3 3]);
 for i = 1: 25
   % read frame i from video
   frame = aviread (filename, i);
   % do frame dilatation
   frame = imdilate (frame, se);
   % write frame
   addframe (writer, frame);
 end

Here, the avifile(filename, frameNo) function reads the frame with the frameNo number from the video file with the name filename . No need to create any objects in advance and open the file. This function is also present in the trademark, but it is deprecated.

If you need to know information about the video, you can do this using the aviinfo(filename) function. As a result, the structure will be returned storing information about the size and format of the video, the number of frames, etc. In the lab, this function is deprecated.

Video is recorded using the avifile and addframe , but the syntax of these functions is slightly different in the hardware and Octave. In addition, in Octave, it is not necessary to close the object recording the video (moreover, there is not even a method that would close it).

As a result, for the above code for Octave to work in the hardware it needs to be modified as follows:

 %% reading video
 filename = 'd: \ test \ result.avi';
 result_file = 'd: \ test \ result2.avi';

 % info = aviinfo (filename);
 % width = info.Width;
 % height = info.Height;

 writer = avifile (result_file, 'compression', 'xvid');
 se = strel ('rectangle', [3 3]);
 for i = 1: 25
    frame = aviread (filename, i);
    frame.cdata = imdilate (frame.cdata, se);
    writer = addframe (writer, frame.cdata);
 end
 writer = close (writer);

Thus, in terms of video and image processing, there is currently a difference in the syntax of Matlabov functions and GNU Octave functions.