Interserver WebRTC

WebRTC can work Peer-to-Peer and Peer-to-Server, where, as a peer, as a rule, a browser or a mobile application. In this article we will talk about how WebRTC works in Server-to-Server mode, what it is for and how it works.

Scaling, Origin-Edge

Where might cross-server WebRTC be needed? The Origin-Edge pattern immediately comes to mind, which is used to scale the broadcast to a large audience.

1. The user sends a WebRTC video stream to the Origin-WebRTC server from a browser or mobile device.
2. Origin server sends stream to multiple Edge servers.
3. Edge servers distribute end users to their browsers and mobile applications.

In modern CDNs, RTMP is actively used for video delivery when publishing a stream to the Origin server and when streaming it to the Edge server, and end users receive the picture via HTTP.

The advantage of WebRTC, compared with this approach, can be a guaranteed low broadcast delay, which cannot be achieved by means of RTMP / HTTP delivery, especially if the nodes are geographically separated.
However, for us, the cross-server WebRTC did not begin with scaling.
')

Load tests

They were always in one form or another. Automatic and semi-automatic, synthetic and close to what users do. We used and used load testing to catch multi-threaded bugs, control resource leaks, optimizations, and many other things that cannot be caught in regular testing.

On the last tests, we raised Linux servers without GUI in the cloud and ran scripts that started many Google Chrome browser processes on the X11 virtual desktop. Thus, real WebRTC browsers were launched that pulled and played WebRTC video streams from the Web Call Server, thereby creating a load as close as possible to the real one. For the server, it looked as if the real user opened the browser and took the video stream, fully utilizing the browser's WebRTC stack, including video decoding and rendering.

The disadvantage of this method of testing was the performance of the test system. It's hard enough to run many Chrome processes on a single Linux system, even when using a powerful server with memory and CPU megameters. As a result, we had to raise a lot of servers and somehow manage / monitor them.

Another limitation was the lack of flexibility - we could not control the processes of chromium. There were only two operations:

1. Raise the process and open the desired URL
2. Kill the process. When the URL was opened, the HTML page was loaded and an automatic connection to the server occurred, already using JavaScript, Websocket + WebRTC. This is how the audience load was modeled.

A flexible load testing tool was required, which would allow giving a load on a server that is close to real, control software testing and ensure high testing performance.

WebRTC Server-Server Testing

We came to the conclusion that the nodes of our server themselves can become load generators, if they are properly hooked to the tested servers.

This idea was implemented in the form of WebRTC pulling . One WCS server can pull a stream from another WCS server via WebRTC. To do this, we introduced the internal abstraction of WebRTCAgent - an object that rises on the testing node and pulls the WebRTC stream from the tested node, connecting to the tested node via Websocket + WebRTC.

After that, we managed WebRTCAgent on REST. As a result, load testing was reduced to calling / pull - methods on the REST interface of the testing node.

When using cross-server WebRTC, we managed to increase the performance of load testing by about 7 times and significantly reduce the use of resources, compared to the scheme when we run Google Chrome processes.

So, we managed to pull WebRTC streams from other servers. The testing server was connected to the test by Websocket, and as a decent browser, set up an ICE connection, DTLS, and pulled SRTP streams on itself, it turned out true WebRTC pulling.

There was very little to get a full-fledged model of the Origin-Edge. For this, it was necessary to forward such pulling to the WCS server engine as a published stream, i.e. make it look like a stream from a webcam, and WCS already knows how to distribute such streams using all available protocols: WebRTC, RTMP, RTMFP, Websocket Canvas, Websocket MSE, RTSP, HLS.

Origin-Edge on WebRTC

It turned out that we did cross-server WebRTC for load testing, but as a result, we implemented a Origin-Edge scheme for scaling WebRTC translations, and here's how it works:

The green line shows how the video traffic passes.

1. A user from a browser or mobile application, using a webcam, sends a WebRTC video stream with the name stream1 to the WCS1 - Origin server. The process of sending a video stream using the Two Way Streaming example web looks like this:

And this is the JavaScript code that is responsible for publishing the video stream through the Web API (Web SDK):

session.createStream({name:'steram1',display:document.getElementById('myVideoDiv')}).publish(); 

2. Next, we access the WCS2 server - Edge via the REST / HTTP API and give the command to pick up the video stream from the Origin server.

 /rest-api/pull/startup { uri: wss://wcs1-origin.com:8443 remoteStreamName: stream1, localStreamName: stream1_edge, } 

The WCS2 server connects via web sockets to the WCS1 server at wss: //wcs1-origin.com: 8443 and receives the stream named stream1 via WebRTC.

After that, you can run a REST command

 /rest-api/pull/find_all 

to output all current pull connections.

Or team

 /rest-api/pull/terminate 

to close a pull connection with the Origin WebRTC server.

3. Finally, pick up the stream from the Edge server via WebRTC in the player . Enter the stream name stream1_edge and start playing it.

WCS server supports several ways to play. To change the technology, simply drag up MSE or WSPlayer to play the stream not via WebRTC, but via MSE (Media Source Extensions) or WSPlayer (Websocket is a player for iOS Safari).

Thus, the Orign-Edge scheme worked and we got the WebRTC server scalability with low latency.

It should be noted that scalability has worked before on RTMP. WCS server can re-publish incoming WebRTC streams to one or several servers using the RTMP protocol. In this case, support for cross-server WebRTC was a step towards reducing the overall system latency.

And again about load testing

For normal load testing, it remains to write a Web interface in the form of a REST client managing pull sessions. This interface was named Console and took the following form:


Using the console, you can pull individual WebRTC streams using the current node as an Edge server. Through the same interface, you can add one or more nodes and run load tests on them with performance evaluation.


There is still much to do and otdebazhit. In particular, it is interesting to work with dynamic bitrates on the cross-server WebRTC channel and compare cross-server patency with RTMP. But now we have Orign-Edge on WebRTC and the right load tests, giving close to real load, which can not but rejoice!

Links

WCS5 - WebRTC Web Call Server 5 Server
Two Way Streaming - an example of video streaming from the browser
WebRTC Player - an example of playing a video stream in a browser with the ability to change technologies WebRTC, MSE, Flash, WSPlayer