Writing a plugin for GStreamer on MS Visual Studio
I have always been interested in real-time video processing applications. On Habré, I read a series of articles on the GStreamer multimedia framework:
I really wanted to do something with its use. But, as usual, the current tasks completely exhausted the resource of free time.
And once, in the course of working on a project, I needed to organize a video surveillance system at the facility and integrate it into the accounting system. This task was successfully solved by our specialists and is not worthy of the attention of the general public. The accounting system works on Microsoft .NET, all cameras issue H264 RTSP stream. Video capture uses the MediaSuite commercial library from Streamcoders.
As a bonus, I allowed myself to conduct a series of experiments with GStreamer for capturing and processing video.
To begin with, I decided to try to capture a stream from one of the cameras aimed at the scales for weighing vehicles, transfer the frame to the license plate recognition subsystem and impose the recognition result on the video. In the above example, I will not call the LPR functional, just grab the frame and draw a rectangle on it.
So, given:
It is necessary to obtain:
Solution:
GStreamer provides the developer with its own class hierarchy . For our purposes, we can inherit our plugin from the GstElement class. GstElement is the base abstract class required to develop an element that can be used in the processing of GStreamer threads.
')
Download and install GStreamer gstreamer-1.0-xxx-1.2.4.msi. It can be taken here .
For development, we also need the distribution gstreamer-1.0-devel-xxx.msi for example, gstreamer-1.0-devel-x86-1.2.4.msi. During the installation process, select the necessary options:
Looking ahead, I will say it would be better to install the Windows Device Driver Kit 7.1.0 . By default, it is installed in C: \ WinDDK \ 7600.16385.1 and this is where Visual Studio will look for it when building the project. If you already have the DDK installed on a different path, this can be corrected later, directly in the project settings.
On the GStreamer project site, among other documentation, there is a guide for plugin developers . You can read the manual to write all the code yourself, but you can download the template for the plugin project from the repository .
The template contains c ++ source files and a script for generating the plugin code. According to the instructions, after deploying the template from the repository, you need to go to the gst-template / gst-plugin / src directory and run the utility ../tools/make_element. The make_element utility has two parameters: the name of the plug-in (dummy), the name of the source file that will be used (gstplugin by default).
As a result of the execution, we get two files: gstdummy.c and gstdummy.h. Inside there will be a skeleton of the dummy plug-in, which is still stupid and does nothing, but can already be built into the system of plug-ins of the framework.
A small remark: everything said above is true for Linux, Unix machines, and how to be mournful owners of Windows? Cmd.exe will not execute make_element. If you look inside make_element, it becomes clear that it doesn’t do anything complicated, and using the stream editor, sed generates target sources based on the parameters it provides. This can be done by yourself. Just in case, I created a repository where I will place my test project in the course of development: github.com/nostrum-service/gst .
After we did the preliminary work, it was time for the project to be formed directly in MS Visual Studio 2010. Fortunately, the GStreamer developers took care of the Visual Studio users and put everything they needed into the distribution to create the project. It is only necessary to correctly place the files in the directories of Visual Studio.
Everything you need is in the directory gstreamer \ 1.0 \ x86 \ share \ vs \ 2010.
We carry out:
We start Visual Studio (or restart it to see the new settings), create a new project, and select Visual C ++ \ gst-dk-template from the installed templates.
If everything went well, an empty project with the necessary settings will be created. Because we want to create a plugin, go to the project settings and change the Project Details Configuration Type from Application (.exe) to Dynamic Library (.dll).
In the Property Manager, we observe the following picture (you can turn on Property Manager- View-> Other Windows-> Property Manager):
The newly created project should include files that were previously created using the make_element utility.
If we compile everything correctly, we get a ready-made DLL that needs to be copied to the GStreamer plugins directory (I have C: \ gstreamer \ 1.0 \ x86 \ lib \ gstreamer-1.0 \).
Just in case, check: gst-inspect-1.0 dummy. Here we will see that GStreamer has learned about our plugin.
In order to embed a plug-in chain, we need to implement the following life cycle:
For providing metadata about the plugin, we have the function
static void gst_dummy_class_init (GstdummyClass * klass)
In our example, the dummy element has a Silent property of type Boolean, which is responsible for displaying text during stream processing.
This code tells the GStreamer environment that the plugin has the Silent property, it has the Boolean type, the delegate is responsible for installing it gst_dummy_set_property, reading is gst_dummy_get_property, it is accessible by reading and writing, the default value is FALSE. Next, we register the connection points to the plugin - pads.
We define the input pad sink, which is always available - GST_PAD_ALWAYS and accepts any format GST_STATIC_CAPS ("ANY").
We define the output src pad, which is always available - GST_PAD_ALWAYS and outputs any format GST_STATIC_CAPS ("ANY").
In the process of building the pipeline, the GStreamer environment calls the plugin instance initialization function:
At the entrance of which is fed the structure, which must be filled:
In order to handle events occurring at the input of the plugin, you must specify the appropriate delegate:
When events occur at the input of the plugin, in our case the function will be called:
which doesn't do anything yet.
Now you can try to run - gst-launch-1.0 videotestsrc! dummy! autovideosink –v. This command sends a test video stream generated by videotestsrc to our plugin, which passes it on to the autovideosink video player without any changes. The v key allows you to see how the entire chain is processed.
If we saw a test image, it means our plugin has successfully transmitted data from its input to the output.
For our case, the output of the processing will contain the following:
From this it follows that an instance of the GstVideoTestSrc class named videotestsrc0 provided a pad src, issuing a video / x-raw stream in I420 format with frame sizes of 320 by 240, and so on.
Since our plugin can accept any format, its input is associated with the output of videotestsrc0:
Having found out that the mechanics works, we will try to take several steps towards our goal. To illustrate the evolution of the solution, I added another painter plug-in to the project that will draw a rectangle over the image.
You can draw a rectangle without using any libraries, but I wanted to continue experimenting using OpenCV.
When adding a new plug-in there is a problem of its registration, since the registrar should be one per project. In this regard, the GstTestLib.cpp file was entered, which contains the registration code for both plugins.
For painter, I created tougher restrictions on the input stream format. Now it looks like this:
This means that BGRx streaming video at any resolution and frame rate can be input. OpenCV uses the BGR scheme by default. About color schemes and transformations can be read here .
Since the camera that interests me produces the RTSP H264 stream, we need to decode it and send it to the input of the videoconvert converter. Our test example will look like this:
(10.10.0.15 - my internal address, taken as an example).
Now the input buffer will be in BGRx format. If you do not know how to build the pipeline, you can use the universal decodebin container. He will try to pick and connect the appropriate plugins:
The –v key is required for diagnostic output, where it will be seen how the pipeline is formed.
Let's return to our plugin. All our processing will consist in forming a new image based on the input and drawing a rectangle on top of it. The working function is:
In the event handler from the sink pad, we can find out information about the input stream.
The problem is solved in the forehead. The experiment was a success. As GStreamer becomes available, new opportunities open up, which I will continue to talk about as I move forward and have an interest in the topic.
GStreamer
GStreamer object hierarchy
GStreamer plugin development guide
Opencv
plugin template repository
my sources
- Introducing GStreamer: Introduction
- Introducing GStreamer: Data Sources
- Meet GStreamer: Output Devices
- GStreamer: Linux-flavored codecs
I really wanted to do something with its use. But, as usual, the current tasks completely exhausted the resource of free time.
And once, in the course of working on a project, I needed to organize a video surveillance system at the facility and integrate it into the accounting system. This task was successfully solved by our specialists and is not worthy of the attention of the general public. The accounting system works on Microsoft .NET, all cameras issue H264 RTSP stream. Video capture uses the MediaSuite commercial library from Streamcoders.
As a bonus, I allowed myself to conduct a series of experiments with GStreamer for capturing and processing video.
To begin with, I decided to try to capture a stream from one of the cameras aimed at the scales for weighing vehicles, transfer the frame to the license plate recognition subsystem and impose the recognition result on the video. In the above example, I will not call the LPR functional, just grab the frame and draw a rectangle on it.
So, given:
- RTSP H264 source
- Windows 7 32-bit system
- MS Visual Studio 2010
- C ++ language
- GStreamer 1.0
It is necessary to obtain:
- video stream containing processing results
Solution:
- Develop plugin for GStreamer 1.0
GStreamer provides the developer with its own class hierarchy . For our purposes, we can inherit our plugin from the GstElement class. GstElement is the base abstract class required to develop an element that can be used in the processing of GStreamer threads.
')
Install GStreamer
Download and install GStreamer gstreamer-1.0-xxx-1.2.4.msi. It can be taken here .
For development, we also need the distribution gstreamer-1.0-devel-xxx.msi for example, gstreamer-1.0-devel-x86-1.2.4.msi. During the installation process, select the necessary options:
Looking ahead, I will say it would be better to install the Windows Device Driver Kit 7.1.0 . By default, it is installed in C: \ WinDDK \ 7600.16385.1 and this is where Visual Studio will look for it when building the project. If you already have the DDK installed on a different path, this can be corrected later, directly in the project settings.
On the GStreamer project site, among other documentation, there is a guide for plugin developers . You can read the manual to write all the code yourself, but you can download the template for the plugin project from the repository .
The template contains c ++ source files and a script for generating the plugin code. According to the instructions, after deploying the template from the repository, you need to go to the gst-template / gst-plugin / src directory and run the utility ../tools/make_element. The make_element utility has two parameters: the name of the plug-in (dummy), the name of the source file that will be used (gstplugin by default).
As a result of the execution, we get two files: gstdummy.c and gstdummy.h. Inside there will be a skeleton of the dummy plug-in, which is still stupid and does nothing, but can already be built into the system of plug-ins of the framework.
A small remark: everything said above is true for Linux, Unix machines, and how to be mournful owners of Windows? Cmd.exe will not execute make_element. If you look inside make_element, it becomes clear that it doesn’t do anything complicated, and using the stream editor, sed generates target sources based on the parameters it provides. This can be done by yourself. Just in case, I created a repository where I will place my test project in the course of development: github.com/nostrum-service/gst .
After we did the preliminary work, it was time for the project to be formed directly in MS Visual Studio 2010. Fortunately, the GStreamer developers took care of the Visual Studio users and put everything they needed into the distribution to create the project. It is only necessary to correctly place the files in the directories of Visual Studio.
Everything you need is in the directory gstreamer \ 1.0 \ x86 \ share \ vs \ 2010.
We carry out:
xcopy c:\gstreamer\1.0\x86\share\vs\2010\gst-template\*.* "C:\Program Files\Microsoft Visual Studio 10.0\VC\VCWizards\gst-template\*.*" /s /e /c xcopy C:\gstreamer\1.0\x86\share\vs\2010\wizard\*.* "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcprojects\" We start Visual Studio (or restart it to see the new settings), create a new project, and select Visual C ++ \ gst-dk-template from the installed templates.
If everything went well, an empty project with the necessary settings will be created. Because we want to create a plugin, go to the project settings and change the Project Details Configuration Type from Application (.exe) to Dynamic Library (.dll).
In the Property Manager, we observe the following picture (you can turn on Property Manager- View-> Other Windows-> Property Manager):
The newly created project should include files that were previously created using the make_element utility.
If we compile everything correctly, we get a ready-made DLL that needs to be copied to the GStreamer plugins directory (I have C: \ gstreamer \ 1.0 \ x86 \ lib \ gstreamer-1.0 \).
Just in case, check: gst-inspect-1.0 dummy. Here we will see that GStreamer has learned about our plugin.
The minimum feature set of the plugin
In order to embed a plug-in chain, we need to implement the following life cycle:
- tell the framework data about yourself (initialize the class)
- initialize the plugin instance
- handle the thread connection process
- do something with the input data and pass it to the output
- on completion, clear occupied resources
Metadata
For providing metadata about the plugin, we have the function
static void gst_dummy_class_init (GstdummyClass * klass)
In our example, the dummy element has a Silent property of type Boolean, which is responsible for displaying text during stream processing.
gobject_class->set_property = gst_dummy_set_property; gobject_class->get_property = gst_dummy_get_property; g_object_class_install_property (gobject_class, PROP_SILENT, g_param_spec_boolean ("silent", "Silent", "Produce verbose output ?", FALSE, (GParamFlags)G_PARAM_READWRITE)); This code tells the GStreamer environment that the plugin has the Silent property, it has the Boolean type, the delegate is responsible for installing it gst_dummy_set_property, reading is gst_dummy_get_property, it is accessible by reading and writing, the default value is FALSE. Next, we register the connection points to the plugin - pads.
gst_element_class_add_pad_template (gstelement_class, gst_static_pad_template_get (&src_factory)); gst_element_class_add_pad_template (gstelement_class, gst_static_pad_template_get (&sink_factory)); We define the input pad sink, which is always available - GST_PAD_ALWAYS and accepts any format GST_STATIC_CAPS ("ANY").
static GstStaticPadTemplate sink_factory = GST_STATIC_PAD_TEMPLATE ("sink", GST_PAD_SINK, GST_PAD_ALWAYS, GST_STATIC_CAPS ("ANY") ); We define the output src pad, which is always available - GST_PAD_ALWAYS and outputs any format GST_STATIC_CAPS ("ANY").
static GstStaticPadTemplate src_factory = GST_STATIC_PAD_TEMPLATE ("src", GST_PAD_SRC, GST_PAD_ALWAYS, GST_STATIC_CAPS ("ANY") ); Instance initialization
In the process of building the pipeline, the GStreamer environment calls the plugin instance initialization function:
static void gst_dummy_init (Gstdummy * filter) At the entrance of which is fed the structure, which must be filled:
struct _Gstdummy { GstElement element; GstPad *sinkpad, *srcpad; gboolean silent; }; In order to handle events occurring at the input of the plugin, you must specify the appropriate delegate:
gst_pad_set_event_function (filter->sinkpad, GST_DEBUG_FUNCPTR(gst_dummy_sink_event)); Connection to stream
When events occur at the input of the plugin, in our case the function will be called:
static gboolean gst_dummy_sink_event (GstPad * pad, GstObject * parent, GstEvent * event) which doesn't do anything yet.
Trial run
Now you can try to run - gst-launch-1.0 videotestsrc! dummy! autovideosink –v. This command sends a test video stream generated by videotestsrc to our plugin, which passes it on to the autovideosink video player without any changes. The v key allows you to see how the entire chain is processed.
If we saw a test image, it means our plugin has successfully transmitted data from its input to the output.
For our case, the output of the processing will contain the following:
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive From this it follows that an instance of the GstVideoTestSrc class named videotestsrc0 provided a pad src, issuing a video / x-raw stream in I420 format with frame sizes of 320 by 240, and so on.
Since our plugin can accept any format, its input is associated with the output of videotestsrc0:
/GstPipeline:pipeline0/Gstdummy:dummy0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive Further immersion
Having found out that the mechanics works, we will try to take several steps towards our goal. To illustrate the evolution of the solution, I added another painter plug-in to the project that will draw a rectangle over the image.
You can draw a rectangle without using any libraries, but I wanted to continue experimenting using OpenCV.
When adding a new plug-in there is a problem of its registration, since the registrar should be one per project. In this regard, the GstTestLib.cpp file was entered, which contains the registration code for both plugins.
struct _elements_entry { const gchar *name; GType (*type) (void); }; static const struct _elements_entry _elements[] = { {"dummy", gst_dummy_get_type}, {"painter", gst_painter_get_type}, {NULL, 0}, }; static gboolean plugin_init (GstPlugin * plugin) { gint i = 0; while (_elements[i].name) { if (!gst_element_register (plugin, _elements[i].name, GST_RANK_NONE, (_elements[i].type) ())) return FALSE; i++; } return TRUE; } For painter, I created tougher restrictions on the input stream format. Now it looks like this:
static GstStaticPadTemplate sink_factory = GST_STATIC_PAD_TEMPLATE ("sink", GST_PAD_SINK, GST_PAD_ALWAYS, GST_STATIC_CAPS ( GST_VIDEO_CAPS_MAKE ("{ BGRx }") ) ); This means that BGRx streaming video at any resolution and frame rate can be input. OpenCV uses the BGR scheme by default. About color schemes and transformations can be read here .
Since the camera that interests me produces the RTSP H264 stream, we need to decode it and send it to the input of the videoconvert converter. Our test example will look like this:
gst-launch-1.0 -v rtspsrc location=rtsp://10.10.0.15 ! rtph264depay ! avdec_h264 ! videoconvert ! painter ! videoconvert ! autovideosink (10.10.0.15 - my internal address, taken as an example).
Now the input buffer will be in BGRx format. If you do not know how to build the pipeline, you can use the universal decodebin container. He will try to pick and connect the appropriate plugins:
gst-launch-1.0 -v rtspsrc location=rtsp://10.10.0.15 ! decodebin ! autovideosink The –v key is required for diagnostic output, where it will be seen how the pipeline is formed.
Let's return to our plugin. All our processing will consist in forming a new image based on the input and drawing a rectangle on top of it. The working function is:
static GstBuffer * gst_painter_process_data (Gstpainter * filter, GstBuffer * buf) { // GstMapInfo srcmapinfo; gst_buffer_map (buf, &srcmapinfo, GST_MAP_READ); // GstBuffer * outbuf = gst_buffer_new (); // IplImage * dst = cvCreateImageHeader (cvSize (filter->width, filter->height), IPL_DEPTH_8U, 4); // GstMemory * memory = gst_allocator_alloc (NULL, dst->imageSize, NULL); GstMapInfo dstmapinfo; if (gst_memory_map(memory, &dstmapinfo, GST_MAP_WRITE)) { // memcpy (dstmapinfo.data, srcmapinfo.data, srcmapinfo.size); dst->imageData = (char*)dstmapinfo.data; // cvRectangle (dst, cvPoint(10,10), cvPoint(100, 100), CV_RGB(0, 255, 0), 1, 0); // gst_buffer_insert_memory (outbuf, -1, memory); gst_memory_unmap(memory, &dstmapinfo); } cvReleaseImageHeader(&dst); gst_buffer_unmap(buf, &srcmapinfo); return outbuf; } In the event handler from the sink pad, we can find out information about the input stream.
static gboolean gst_painter_sink_event (GstPad * pad, GstObject * parent, GstEvent * event) { gboolean ret; Gstpainter *filter; filter = GST_PAINTER (parent); switch (GST_EVENT_TYPE (event)) { case GST_EVENT_CAPS: { GstCaps * caps; gst_event_parse_caps (event, &caps); // GstStructure *structure = gst_caps_get_structure (caps, 0); // gst_structure_get_int (structure, "width", &filter->width); gst_structure_get_int (structure, "height", &filter->height); filter->format = gst_structure_get_string (structure, "format"); ret = gst_pad_event_default (pad, parent, event); break; } default: ret = gst_pad_event_default (pad, parent, event); break; } return ret; } The problem is solved in the forehead. The experiment was a success. As GStreamer becomes available, new opportunities open up, which I will continue to talk about as I move forward and have an interest in the topic.
Sources
GStreamer
GStreamer object hierarchy
GStreamer plugin development guide
Opencv
plugin template repository
my sources
Source: https://habr.com/ru/post/221483/
All Articles