We broadcast WebRTC, RTSP and RTMP streams to Media Source Extensions via Websocket protocol

Media Source Extensions

Media Source Extensions (hereinafter MSE) is a browser API that allows you to play audio and video through the corresponding HTML5 tags <audio /> and <video />.

To play a piece of audio or video, you need to feed this chunk to this element through the MSE API. Based on MSE built HLS-players. HLS fragments are transmitted to MSE and displayed in the player.

Let's take a look at its Can I Use for details.
')

As the table shows, MSE is supported in all recent browsers except iOS Safari and Opera Mini. In this regard, it is very similar to WebRTC, and even bypasses WebRTC for accessibility in Safari and IE. I must say that Safari 11 already has WebRTC, but it has not yet been released at the time of this writing.


It should be noted that MSE and WebRTC are technologies of different weight categories. Roughly speaking, MSE is only a player, and WebRTC is both a player and a streamer and calls (low latency real-time streaming).

Thus, if you need only a player and no real-time requirements (delays less than 1 second), then MSE is a good choice for playing video streams. If you need real-time and without plug-ins, then there are no options. Need to take WebRTC.

Media Source Extensions	Webrtc
Player only	Player and Streamer
Average delay	Realtime (delay less than 1 second)

WebRTC to MSE

Suppose WebRTC acts as a tape drive. This means that the browser or mobile application sends a video stream to the server.

The fastest way to pick up a video stream in MSE is to connect to the server using the Websocket protocol and deliver the stream to the browser. In the player, parse the received stream and transfer it to MSE for playback.

Broadcast scheme:

1. The browser of the broadcaster sends the stream H.264 + Opus to the WebRTC server
2. WebRTC server distributes stream via Websocket H.264 + AAC protocol
3. The viewer browser opens the stream and gives H.264 and AAC frames to play back in MSE.
4. The player plays audio and video.

Thus, when using Media Source Extensions as a player, the video part of the WebRTC stream with the H.264 codec passes to the player without transcoding (proxied), which helps keep CPU utilization low on the server.

The Opus audio codec is transcoded into AAC for successful reading in MSE, but audio transcoding is much less resource intensive than video.

Examples

These examples of tape drive and MSE player use Web Call Server 5 as a server that converts a WebRTC video stream for use in Media Source Extensions.

As a streamer, we use the Two Way Streaming web application , which sends a WebRTC video stream to the server.


As a player, we use a test player with support for Websockets and MSE.


The MSE player code works through a high-level API.

Create a div element on the HTML page

<div id="myMSE" style="width:640;height:360"></div> 

And then call the playback of the video stream.

Passing the div to the element with the display parameter and the MSE player will be inserted into this div block.

 session.createStream({name:"stream22",display:document.getElementById("myMSE")}).play(); 

Similarly, publishing a WebRTC stream from another page works.

The stream is assigned the unique name stream22, which will be specified during playback.

 session.createStream({name:"stream22",display:document.getElementById("myWebCam")}).publish(); 

The full code of the tape drive and player can be found on github.
- streamer
- player

RTSP to MSE

Another case of using MSE over Websockets is to play video from an IP camera or another system that sends a video stream over RTSP.

An IP camera, as a rule, natively supports H.264 and AAC codecs, so the codecs completely coincide with those used on MSE. This helps avoid transcoding, which consumes CPU resources.

The broadcast scheme is as follows:

1. Browser requests RTSP stream.
2. The server establishes a connection with the camera and requests this stream from the camera via RTSP.
3. The camera sends an RTSP stream. Streaming begins.
4. The RTSP stream is converted to Websockets on the server side and down to the browser.
5. The browser sends the stream to the MSE player for playback.

To work with the RTSP stream, use the same player that we showed above. Just set the full URL of the RTSP stream instead of the name.

At the time of this writing, we have tested the MSE player in Chrome, Firefox, Safari.

RTMP to MSE

RTMP is actually the standard for delivering live content from a user to a CDN.

Therefore, you need to make sure that standard RTMP streams with H.264 and AAC codecs play correctly in Media Source Extensions.

This can be quickly checked using a wirecast encoder (alternatively, you can use the FMLE, ffmpeg, OBS encoder, and others).

1. Open Wirecast and the Output menu. Set the server address and the name of the RTMP stream as stream44.


2. Stream stream to the server.


3. Play the stream in the MSE-player.

Comparison with other playback technologies

Now let's take a look at MSE in comparison with other video playback technologies in the browser.

For comparison, take 4 technologies:

• Webrtc
• Flash
• Hls
• Canvas + Web Audio

	MSE	Webrtc	Flash	Hls	Canvas
Chrome, Firefox support	Yes	Yes	Yes	Yes	Yes
IOS Safari support	Not	Not	Not	Yes	Yes
Mac Safari Support	Yes	Not	Yes	Yes	Yes
Delay	3	0.5	3	20	3
Full HD resolution	Yes	Yes	Yes	Yes	Not

Thus, if you need to play the live stream in a browser with a slight delay, MSE is a good solution, replacing Flash in the latest browsers.

If the delay is absolutely not important, you should use HLS, as the most universal option.

If low latency is critical, you need to use a combination of WebRTC, Flash, Canvas, MSE to cover most browsers.

The delay should be less than 1 second and without plug-ins, use WebRTC.

Links

Two Way Streaming - demo webRTC streaming to server
Source - source code of the WebRTC stream stream

Player - demo player with support for Media Source Extension
Source - MSE support source code

Web Call Server 5 - a server that supports WebRTC and RTSP broadcasts with stream conversion for Media Source Extensions

Can I Use MSE - browsers that support MSE
Can I Use WebRTC - browsers that support WebRTC