GStreamer: Linux-flavored codecs

Have you ever thought about how Gnome players work, such as Totem, Rhythmbox or Banshee? Probably each of you in the newly installed Ubuntu, when trying to play AVI-shku, saw a message about the need to install an additional package gst-ffmpeg or gst-plugins-ugly. Under the cut - my attempt to clarify the light on how this multimedia framework works and what it can do.

The main difference from VLC or FFmpeg is that the environment was originally developed not for a specific application, but for all programs that can work with multimedia. This framework is integrated into the following languages: C, C ++ / Qt, Python, Ruby, Perl.

GStreamer allows you to create applications that can:

• Receive and transmit audio video using almost any protocol: HTTP, RTSP / RTP, MMS, RTMP and others.
• Disassemble / Collect streams in different formats: MPEG, AVI, ASF, FLV, MKV and others.
• Pull and embed metadata in files.
• Decode / Encode streams in almost any combination of codecs.
  In this case, you can use not only open-source solutions (libavcodec, libmp3lame, libfaac), but also proprietary solutions that you can buy. It is also possible to use decoders with hardware support using CUDA / VAAPI / CrystalHD technologies.
• Display streams on the screen or sound card, using a variety of solutions, ranging from DirectFB, and ending X-s.

To solve problems such as transcoding a movie into another format, such programs as mencoder or VLC are very popular. But doing the same thing with GStreamer is not much more difficult.

Below are examples for Linux that demonstrate the capabilities of the framework. In these examples, we will use the main programs gst-launch and gst-inspect — somewhat reminiscent of the Windows application GraphEdit. For those who need a graphical shell, there is gst-editor . But keep in mind that these programs are designed for debugging, so do not expect ease of use from them.
')
To get started with GStreamer, you need to understand some of the concepts:

• element - A GStreamer element object that can receive / send data.
• source - Generates a stream (file, stream or device).
• filter - An element that transforms a stream (parsers, multiplexers, codecs).
• sink - Accepts a stream and does not redirect it.
• pipeline - A collection of interconnected elements.

In the classic form, we get this kind of chain: source -> parser -> decoder -> filter -> sink.

If you are too lazy to assemble the whole chain of codecs and multiplexers manually, GStreamer can do it for you:

$ gst-launch playbin uri=file:///home/me/audo.mp3

The playbin element itself parses the file, finds the necessary codec and output element. All you need is to specify the path to the file you want to play. If you are wondering which element is responsible for what, you can run gst-inspect and see all available elements, and “gst-inspect lame” will show the parameters of the mp3 coder. It is also worth noting the usefulness of the “decodebin2” element, which itself finds the desired decoder.

The simplest audio player:

$ gst-launch filesrc location=audio.mp3 ! decodebin2 ! alsasink

The simplest audio player (choose the codec manually, and agree on the formats):

$ gst-launch filesrc location=audio.mp3 ! mad ! audioconvert ! audioresample ! alsasink

The player is more complicated (with sound and video):

$ gst-launch filesrc location=audovideofile.mpeg ! decodebin2 name=decoder \
decoder. ! ffmpegcolorspace ! xvimagesink \
decoder. ! audioconvert ! audioresample ! alsasink

If you need to transcode an AVI file into MOV, you can use the following command:

$ gst-launch-0.10 filesrc location=film.avi ! \
decodebin2 name=decoder { qtmux name=muxer ! filesink location=film.mov } \
{ decoder. ! ffmpegcolorspace ! jpegenc ! queue ! muxer. } \
{ decoder. ! queue ! audioconvert ! queue ! muxer. }

I want to make a broadcast, simultaneously with recording to a file and displaying the broadcast video on the screen? No problem:

$ gst-launch v4l2src ! queue ! \
ffmpegcolorspace ! tee name=t1 ! \
queue ! xvimagesink sync=false t1. ! \
queue ! theoraenc quality=1 ! \
queue ! oggmux name=mux \
alsasrc ! audio/x-raw-int,rate=8000,channels=1,depth=8 ! \
queue ! audioconvert ! vorbisenc ! \
queue ! mux. mux. ! \
queue ! tee name= t ! \
queue ! filesink location=test.ogg t. ! \
queue ! shout2send ip=<-icecast-> port=8000 password=hackme


Fans of visiting in a flash - also did not stand aside, thanks to librtmp (you need to rebuild the package gst-plugins-bad):

$ gst-launch v4l2src ! \
queue ! ffmpegcolorspace ! ffenc_flv ! \
queue ! flvmux name=muux is-live=true ! \
rtmpsink location='rtmp://localhost/path/to/stream live=1' \
alsasrc ! audio/x-raw-int,rate=22050,channels=1,depth=16 ! \
queue ! audioconvert ! lame bitrate=48 ! \
queue ! muux.

Also, the developers took care of the use of "native" multimedia frameworks (QuickTime / DirectShow), if you are using MacOS or Windows, thanks to the "bridges".

Well, in conclusion, I would like to say that there are a lot of problems that developers have yet to solve (desync, problems with memory leaks and performance). Most of them are related to the use of third-party libraries, which are not always well written. That's why the developers came up with an interesting trick - the plug-ins are divided into 5 categories:

• gst-plugins-base - The main plug-ins that come with: CDDA, RTP, RTSP, ogg, theora, vorbis, Xvideo, V4L.
• gst-plugins-good - Plugins that are not in doubt about their reliability: FLAC, speex, JPEG, PNG.
• gst-plugins-bad - Slaves, but problems are possible: VP8, XVID, FAAC, dirac, RTMP, VDPAU, DirectShow.
• gst-plugins-ugly - Not to be without problems when using: ASF, real, DVDSub, a52, AMR, x264.
• gst-ffmpeg - A separate set of all-in-one codecs.

Mistress of the note (sources):

1. GStreamer features
2. Live stream with GStreamer examples for gst-launch
3. Howto
4. GStreamer, RTP and live streaming , examples for python
5. Collections of paid codecs
6. Hello World GStreamer in C