How to get a neighbor or Karaoke on GStreamer

We continue to master the GStreamer framework. As promised, today we will consider in more detail the work with the bus and the processing of messages of various types. And in the practical section, we will write a small console karaoke. So, let's begin.

Tire

Let me remind you that the application on GStreamer consists of various elements placed in a conveyor and interconnected. Each element in the course of its work can generate messages of different types (described below). A bus is used to collect these messages and deliver them in turn to the main application flow.

Each pipeline has a built-in bus by default, i.e. it does not need to be created or customized. The tire is removed from the conveyor by the following function:
')

GstBus * gst_pipeline_get_bus (GstPipeline *pipeline) 

There are two ways to process messages on the bus - asynchronous and synchronous. The first way is that a handler is connected to the bus, which after receiving a message performs some actions. This is done using functions:

 guint gst_bus_add_watch (GstBus *bus, GstBusFunc func, gpointer user_data) void gst_bus_add_signal_watch (GstBus *bus) 

It is worth noting that both of these functions have advanced options (* _full), with the help of which a more fine-tuning of the handler is possible.
The second way is to constantly poll the bus for messages. This is done using the function:

 GstMessage *gst_bus_poll (GstBus *bus, GstMessageType events, GstClockTime timeout) 

At the same time, it is possible to take only certain types of messages from the queue, as well as to adjust the duration of the poll.

Messages

Each message is characterized by its source, type, and time stamp. Most often, there is no need to respond to each type of message, but it is highly desirable to handle at least error messages. There are a lot of types, here are the most common:

• Errors, warnings and informational messages
• End of Stream messages
• Tags
• State changes

You can approach the issue of processing messages on the other hand, namely, to connect only to the signals of interest to us in the following form:

 g_signal_connect (bus, "message::error", G_CALLBACK (cb_message_error), NULL) 

Important! If you want the handler to continue to respond to messages, it must return TRUE. If, however, to register the return FALSE, then after processing the tracking of messages by this handler will stop.

Practice

Now let's write an application that plays an mp3 file and suppresses a certain frequency range (which includes the vocal range) in it using the audiokaraoke element. In this case, we get the track without vocals. Also the text of the song from the file (in UTF-8 encoding) is output to the console.

The conveyor will look like this:



Make sure you have the Ugly plug-in set (gst-plugins-ugly) installed, since it includes Mad mp3 decoder.

And here is the program itself:

 #include <gst/gst.h> #define SUPRESS_LEVEL 1.0 //  0.0  1.0 #define MONO_LEVEL 1.0 //  0.0  1.0 #define FILTER_BAND 220.0 //  0.0  441.0 #define FILTER_WIDTH 100.0 //  0.0  100.0 static GMainLoop *loop; static gboolean my_bus_callback (GstBus *bus, GstMessage *message, gpointer data) { switch (GST_MESSAGE_TYPE (message)) { case GST_MESSAGE_ERROR: { GError *err; gchar *debug; gst_message_parse_error (message, &err, &debug); g_print ("Error: %s\n", err->message); g_error_free (err); g_free (debug); g_main_loop_quit (loop); break; } case GST_MESSAGE_EOS: { g_print ("\n\nYou've got 100 points!\n"); g_main_loop_quit (loop); break; } default: break; } /*  TRUE,   ,       */ return TRUE; } int main (int argc, char * argv[]) { if (argc != 3) { g_printerr("Usage: %s audiofile.mp3 lyricsfile.txt\n", argv[0]); return -1; } GstElement *pipeline, *audiosrc, *parser, *decoder, *converter, *karaoke, *audiosink; GstElement *textsrc, *textsink; GstStateChangeReturn ret; GstBus *bus; guint bus_watch_id; /*  */ gst_init(NULL, NULL); /*   */ pipeline = gst_element_factory_make ("pipeline", "pipe"); audiosrc = gst_element_factory_make ("filesrc", "audiosrc"); parser = gst_element_factory_make ("mpegaudioparse", "parser"); decoder = gst_element_factory_make ("mad", "decoder"); converter = gst_element_factory_make ("audioconvert", "converter"); karaoke = gst_element_factory_make ("audiokaraoke", "karaoke"); audiosink = gst_element_factory_make ("autoaudiosink", "audiosink"); textsrc = gst_element_factory_make ("filesrc", "textsrc"); textsink = gst_element_factory_make ("fdsink", "textsink"); if (!pipeline || !audiosrc || !parser || !decoder || !converter || !karaoke || !audiosink || !textsrc || !textsink) { g_printerr ("Unable to create some elements\n"); return -1; } /*     */ gst_bin_add_many (GST_BIN(pipeline), audiosrc, parser, decoder, converter, karaoke, audiosink, textsrc, textsink, NULL); /*    */ if (gst_element_link_many (audiosrc, parser, decoder, converter, karaoke, audiosink, NULL) != TRUE) { g_printerr ("Unable to link some elements\n"); gst_object_unref(pipeline); return -1; } if (gst_element_link (textsrc, textsink) != TRUE) { g_printerr ("Unable to link text with textsink\n"); gst_object_unref(pipeline); return -1; } /*    */ g_object_set (audiosrc, "location", argv[1], NULL); g_object_set (textsrc, "location", argv[2], NULL); g_object_set (karaoke, "level", SUPRESS_LEVEL, "mono-level", MONO_LEVEL, "filter-band", FILTER_BAND, "filter-width", FILTER_WIDTH, NULL); /*   */ ret = gst_element_set_state (pipeline, GST_STATE_PLAYING); if ( ret == GST_STATE_CHANGE_FAILURE ) { g_printerr ("Unable to set pipeline to the playing state\n"); gst_object_unref (pipeline); return -1; } /*          */ bus = gst_element_get_bus (pipeline); bus_watch_id = gst_bus_add_watch (bus, my_bus_callback, NULL); gst_object_unref (bus); /*     */ loop = g_main_loop_new (NULL, FALSE); g_main_loop_run (loop); /*   */ gst_element_set_state (pipeline, GST_STATE_NULL); gst_object_unref (pipeline); g_source_remove (bus_watch_id); g_main_loop_unref (loop); return 0; } 

We compile as usual

 $ gcc -Wall -o karaoke karaoke.c $(pkg-config --libs --cflags gstreamer-1.0) 

and run

 $ ./karaoke audiofile.mp3 lyricsfile.txt 

results

I tried to run the application (with the default filter settings) on the following songs:

Song	Effect
Tool - Parabola	Not bad cut vocals, but suppressed a lot of instrumental (especially the flageoles in solo)
At the Drive-In - Rolodex Propaganda	The instrumental is well preserved, but the vocals are quite strong (apparently, due to its height)
Radiohead - Jigsaw Falling into Place	It turned out pretty interesting here - the main vocal was crushed, and the backing remained
U2 - Raised by Wolves	The guitar is kept almost perfect.

It is clear that the result strongly depends on the settings of the filter and on the song itself, so you can play with its parameters specified via #define and see what happens.