Java, Spring, Kurento and media services. Part 2

In the previous article, I described my experiment with the “HelloWorld” application from Kurento-tutorial. In this article, I will continue to review Java Spring and Kurento Media Server Web applications.

The following app from the tutorial is called WebRTC Magic Mirror. From the name it follows that the mirror reflects a slightly embellished reality. In this application, in addition to the previous one, the computer vision filter “FaceOverlay filter” is applied. When a person's face is detected in a camera, Kurento Media Server “puts on” a Super Mario cap on his head. If there are several faces in the objective, each one is wearing a cap.

Launch Kurento Media Server

sudo service kurento-media-server start 

If you have already cloned the Kurento Java tutorial, do not run the following command:
')

 git clone https://github.com/Kurento/kurento-tutorial-java.git 

Go to the project directory

 cd kurento-tutorial-java/kurento-magic-mirror 

We compile the project with the Maven team.

 mvn compile exec:java 

Or, as in my case, let's launch from Idea.

In the Firefox browser, go to https: // localhost: 8443 /

In the window that opens, click on the "Start" button, you should see the picture:


From this experience we conclude: the muzzle of my cat is not at all like a human, so the Super Mario cap is not fit for him.

Let us see how this example works.

The structure of the application repeats the previous HelloWorld application. Three software blocks interact in the application: on the browser side, the Java Script code, on the server side of the Spring-Boot Web application, which controls the Kurento Media Server. These program blocks exchange messages based on the Json protocol with each other.



The diagram shows that in the KMS the WebRtcEndpoint program block no longer wraps the transmission of the multimedia signal to the reception, but rather passes it through FaceOverlayFilter, which determines the presence of a face in the picture.

The exchange of signals between software blocks is about the same as in the HelloWorld application.



The main actions in the application are performed when you click on the "Start" button. A MediaPipeline is created, through which the media traffic will be transmitted, the WebRtcEndpoint and FaceOverlayFilter media elements are created, and a connection between them is created.

Here is a simplified sequence of statements that forms a chain of program blocks for a given application.

 MediaPipeline pipeline = kurento.createMediaPipeline(); WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(pipeline).build(); webRtcEndpoint.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); 

Consider the following application from Kurento Tutorial called RTP Receiver.

If not running, start Kurento Media Server

 sudo service kurento-media-server start 

If you have already cloned the Kurento Java tutorial, do not run the following command:

 git clone https://github.com/Kurento/kurento-tutorial-java.git 

Go to the project directory

 cd kurento-tutorial-java/kurento-rtp-receiver 

We compile the project with the Maven team.

 mvn compile exec:java 

or, as in my case, we will launch from Idea.

In the Firefox browser, go to https: // localhost: 8443 /

in the window that opens, click on the "Start" button



In the KMS connection info window, the data for setting the command for Gstreamer appears: audio, video port, IP address. This data, as well as the file path must be specified in the command.

 PEER_A={KMS_AUDIO_PORT} PEER_V={KMS_VIDEO_PORT} PEER_IP={KMS_PUBLIC_IP} \ SELF_PATH="{PATH_TO_VIDEO_FILE}" \ SELF_A=5006 SELF_ASSRC=445566 \ SELF_V=5004 SELF_VSSRC=112233 \ bash -c 'gst-launch-1.0 -e \ rtpbin name=r sdes="application/x-rtp-source-sdes,cname=(string)\"user\@example.com\"" \ filesrc location="$SELF_PATH" ! decodebin name=d \ d. ! queue ! audioconvert ! opusenc \ ! rtpopuspay ! "application/x-rtp,payload=(int)96,clock-rate=(int)48000,ssrc=(uint)$SELF_ASSRC" \ ! r.send_rtp_sink_0 \ d. ! queue ! videoconvert ! x264enc tune=zerolatency \ ! rtph264pay ! "application/x-rtp,payload=(int)103,clock-rate=(int)90000,ssrc=(uint)$SELF_VSSRC" \ ! r.send_rtp_sink_1 \ r.send_rtp_src_0 ! udpsink host=$PEER_IP port=$PEER_A bind-port=$SELF_A \ r.send_rtcp_src_0 ! udpsink host=$PEER_IP port=$((PEER_A+1)) bind-port=$((SELF_A+1)) sync=false async=false \ udpsrc port=$((SELF_A+1)) ! r.recv_rtcp_sink_0 \ r.send_rtp_src_1 ! udpsink host=$PEER_IP port=$PEER_V bind-port=$SELF_V \ r.send_rtcp_src_1 ! udpsink host=$PEER_IP port=$((PEER_V+1)) bind-port=$((SELF_V+1)) sync=false async=false \ udpsrc port=$((SELF_V+1)) ! tee name=t \ t. ! queue ! r.recv_rtcp_sink_1 \ t. ! queue ! fakesink dump=true async=false' 

By the way, check if you have Gstreamer installed.

 gst-launch-1.0 —gst-version 

They substituted ports and stuff, they launched the command Gstreamer-a in the console.

Now, attention play the lottery. I do not know whether or not you will play the file. I started to play files with the extension * .mpg from my poor collection of clips



Here is a clip of Natalia Mogilev. Clips with a different extension are either not played, or are played with sound distortion.

From this experience we conclude: not all yogurts are equally useful.

We analyze the work of this application. First, Media Pipeline is created to transfer media traffic, as well as RtpEndpoint and WebRtcEndpoint, the latter performs the work of the RTP packet receiver and is used to display the video clip.

Here is a simplified sequence of statements for this part of the Java application:

 final MediaPipeline pipeline = kurento.createMediaPipeline(); final RtpEndpoint rtpEp = makeRtpEndpoint(pipeline, useSrtp); final WebRtcEndpoint webRtcEp = new WebRtcEndpoint.Builder(pipeline).build(); rtpEp.connect(webRtcEp); 

This concludes the overview of this part of the Kurento tutorial. Happy programming!

Kurento
Kurento tutorial
Previous article