Debugging and optimization of gstreamer scripts using live streaming as an example

Gstreamer is the most popular * nix video library, which is the basis for most popular applications (the main list can be found here ).



However, such a flexible tool requires a subtle approach for stable and efficient operation.
All actions will be discussed on the example of online HD broadcast from the HDMI input via RTMP using blackmagic intensity pro, Debian Linux, gstreamer and Nginx.

Debugging ( manual )

Gstreamer has a built-in debugging feature; you can get help with the command:

gst-launch --gst-debug-help 

The very output of debug information is included by adding to the command:

 --gst-debug-level=LEVEL 

where LEVEL is a number from 0 (without debugging output) to 9 (display all).
')
A very useful parameter is

 --gst-debug=GST_CAPS:4 

, which will display a list of possible formats of elements of the conveyor, if two elements cannot find suitable formats for each other.

However, the main problem is the huge amount of information that needs to be interpreted in some way for the purpose of further optimization.
In this situation, there is a great opportunity to generate gstreamer pipeline schemes, which will allow you to visually see the working order of your pipeline:

• To generate a .DOT file and further conversion, you need to install the graphviz package.
  

   apt-get install graphviz 

• Next, you need to create an environment variable:
  

   export GST_DEBUG_DUMP_DOT_DIR=/tmp/dot mkdir -p $GST_DEBUG_DUMP_DOT_DIR 

• Run Gstreamer with your pipeline, .dot files will be created in the / tmp / dot folder for each status of the pipeline.
• Convert the resulting files to a convenient format for you:

   dot -Tpng input.dot > output.png dot -Tsvg input.dot > output.svg 

  Using SVG will allow infinitely scale the resulting scheme, since there is enough information on it.
  Example schema.

According to the data obtained, it will be possible to find the bottlenecks of your pipeline and achieve stable operation.

Optimization

As part of the optimization example, the following requirements will be considered:

• 1920x1080 Full HD stream resolution
• 30 frames per second
• H264 codec with a bitrate of 768kbit / s
• flv rtmp container

The goal is to operate the system in real time without loss of frames and synchronize audio / video.

Frames / sec ( fps )

It is very important to immediately find out how many frames per second is required and how much your equipment can deliver.
Optimal for a dynamic image is 30fps, but for transferring static content (for example, presentations), it makes sense to reduce fps to the least comfortable one.

Multithreading (using queue )

In Gstreamer, there is an element queue that justifies its name - it is a queue (stream) of data.
Using queue allows you to parallelize calculations and buffer information in automatic mode for transmission between elements.
This element is one of the key along with queue2 (analogue queue with buffering on the hard disk) and tee (splitting the stream into several, for example, to write the stream to disk in parallel), and they all deserve a separate article.

Configuring conveyor elements

Of course, the key point is to set up the elements of the pipeline (audio and video decoder, etc.).

• Avoid copying buffers whenever possible.

  v4l2src always-copy = false
  filesink enable-last-buffer = false

• Setting the video encoder to work in real time
  Critical for online broadcasts
  

  x264enc tune = zerolatency
  dmaienc_h264 encodingpreset = 2

  
  
• Using variable bitrate for fast encoder operation
  Performance benefit - up to 3 times
  

  x264enc pass = 17
  dmaienc_h264 ratecontrol = 2

  
  

Practice

An example of an optimized minimum pipeline:

 gst-launch \ audiotestsrc ! queue ! audioresample ! voaacenc bitrate=64 ! audio/mpeg,rate=22050,channels=1 ! \ flvmux streamable=1 name=mux \ videotestsrc ! queue ! videorate ! videoscale ! x264enc bitrate=768 tune=zerolatency pass=17 ! \ mux. mux. ! \ rtmpsink location="rtmp://localhost/live/test" 

In this case, we are preparing a stream with a video bitrate of 768 and one channel audio bitrate of 64 kbit / s, respectively, using queue and converting video with a variable bit rate.
See the scheme of the conveyor here .

The above example does not take into account some of the requirements of the task, the final pipeline:

 gst-launch \ audiotestsrc ! queue ! audioresample ! audio/x-raw-int,width=16,depth=16,channels=2,rate=22050 ! \ audioconvert ! voaacenc bitrate=64 ! audio/mpeg,rate=22050,channels=1 ! \ flvmux streamable=1 name=mux \ videotestsrc ! queue ! videorate ! videoscale ! ffmpegcolorspace ! \ 'video/x-raw-yuv, format=(fourcc)UYVY, framerate=(fraction)30/1' ! \ ffmpegcolorspace ! x264enc bitrate=768 tune=zerolatency pass=17 ! \ mux. mux. ! \ rtmpsink location="rtmp://localhost/live/test" 

In this pipeline, it is assumed that videotestsrc gives 1080p.
This example is real and is used on a server with a 16 core processor and a Blackmagic intensity pro pci board, the total load is 0.5 LA.
In both examples, the web server used Nginx with a great user module rarutyunyan nginx-rtmp-module

Of course, there are many other tools and features for debugging and optimizing Gstreamer,
I will be glad to your examples!

Continuous broadcasts to you, colleagues!

Sources and references:

• The gstreamer debug info features can be found here .
• GStreamer Debugging
• Performance Optimization Embedded Solutions
• Gstreamer official website