Geekly Articles each Day
📜 ⬆️ ⬇️

We develop video chat between the browser and the mobile application


Evil empires often receive hate rays from end users. In spite of this, Uber partially pays for our trips, albeit temporarily, and Google gave a significant acceleration of the WebRTC technology, which would have remained proprietary and heavily paid software for narrow b2b goals, if not for IZ .

After the advent of WebRTC, video chats became easier to do. Various APIs and services appeared, servers and frameworks. In this article, we describe in detail another way to develop video chat between a web browser and a native Android application.

Video chat in browser


Classic WebRTC video chat between browsers begins with the SDP (session description protocol) exchange. Alice sends her SDP over the network to Boris , and Boris answers hers. SDP, which is such a config:
')
o = - 1468792194439919690 2 IN IP4 127.0.0.1
s = -
t = 0 0
a = group: BUNDLE audio video
a = msid-semantic: WMS 9nKsWmxMvOQBYaz9xhRffWeWSUbCbnox6aQ4
m = audio 9 UDP / TLS / RTP / SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
c = IN IP4 0.0.0.0
a = rtcp: 9 IN IP4 0.0.0.0
a = ice-ufrag: kSrQ
a = ice-pwd: 4uIyZd / mbVSVe2iMFgC1i3qn
a = fingerprint: sha-256 6B: 29: 2F: 47: EB: 38: 64: F3: 25: CE: BD: E6: B0: 3F: A6: FA: 55: 57: A9: EA: 44: 0B : 7C: 45: D2: 0D: F4: 96: 8D: B2: 9F: BA
a = setup: actpass
a = mid: audio
a = extmap: 1 urn: ietf: params: rtp-hdrext: ssrc-audio-level
a = sendonly
a = rtcp-mux
a = rtpmap: 111 opus / 48000/2
a = rtcp-fb: 111 transport-cc
a = fmtp: 111 minptime = 10; useinbandfec = 1
a = rtpmap: 103 ISAC / 16000
a = rtpmap: 104 ISAC / 32000
a = rtpmap: 9 G722 / 8000
a = rtpmap: 0 PCMU / 8000
a = rtpmap: 8 PCMA / 8000
a = rtpmap: 106 CN / 32000
a = rtpmap: 105 CN / 16000
a = rtpmap: 13 CN / 8000
a = rtpmap: 110 telephone-event / 48000
a = rtpmap: 112 telephone-event / 32000
a = rtpmap: 113 telephone-event / 16000
a = rtpmap: 126 telephone-event / 8000
a = ssrc: 3525514540 cname: drYey7idt605CcEG
a = ssrc: 3525514540 msid: 9nKsWmxMvOQBYaz9xhRffWeWSUbCbnox6aQ4 09bdb6e7-b4b3-437b-945e-771f535052e3
a = ssrc: 3525514540 mslabel: 9nKsWmxMvOQBYaz9xhRffWeWSUbCbnox6aQ4
a = ssrc: 3525514540 label: 09bdb6e7-b4b3-437b-945e-771f535052e3
m = video 9 UDP / TLS / RTP / SAVPF 96 98 100 102 127 97 99 101 125
c = IN IP4 0.0.0.0
a = rtcp: 9 IN IP4 0.0.0.0
a = ice-ufrag: kSrQ
a = ice-pwd: 4uIyZd / mbVSVe2iMFgC1i3qn
a = fingerprint: sha-256 6B: 29: 2F: 47: EB: 38: 64: F3: 25: CE: BD: E6: B0: 3F: A6: FA: 55: 57: A9: EA: 44: 0B : 7C: 45: D2: 0D: F4: 96: 8D: B2: 9F: BA
a = setup: actpass
a = mid: video
a = extmap: 2 urn: ietf: params: rtp-hdrext: toffset
a = extmap: 3 www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a = extmap: 4 urn: 3gpp: video-orientation
a = extmap: 5 www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a = extmap: 6 www.webrtc.org/experiments/rtp-hdrext/playout-delay
a = sendonly
a = rtcp-mux
a = rtcp-rsize
a = rtpmap: 96 VP8 / 90000
a = rtcp-fb: 96 ccm fir
a = rtcp-fb: 96 nack
a = rtcp-fb: 96 nack pli
a = rtcp-fb: 96 goog-remb
a = rtcp-fb: 96 transport-cc
a = rtpmap: 98 VP9 / 90000
a = rtcp-fb: 98 ccm fir
a = rtcp-fb: 98 nack
a = rtcp-fb: 98 nack pli
a = rtcp-fb: 98 goog-remb
a = rtcp-fb: 98 transport-cc
a = rtpmap: 100 H264 / 90000
a = rtcp-fb: 100 ccm fir
a = rtcp-fb: 100 nack
a = rtcp-fb: 100 nack pli
a = rtcp-fb: 100 goog-remb
a = rtcp-fb: 100 transport-cc
a = fmtp: 100 level-asymmetry-allowed = 1; packetization-mode = 1; profile-level-id = 42e01f
a = rtpmap: 102 red / 90000
a = rtpmap: 127 ulpfec / 90000
a = rtpmap: 97 rtx / 90000
a = fmtp: 97 apt = 96
a = rtpmap: 99 rtx / 90000
a = fmtp: 99 apt = 98
a = rtpmap: 101 rtx / 90000
a = fmtp: 101 apt = 100
a = rtpmap: 125 rtx / 90000
a = fmtp: 125 apt = 102
a = ssrc-group: FID 2470936840 2969787502
a = ssrc: 2470936840 cname: drYey7idt605CcEG
a = ssrc: 2470936840 msid: 9nKsWmxMvOQBYaz9xhRffWeWSUbCbnox6aQ4 ce9235c5-f300-466a-aadd-b969dc2f3664
a = ssrc: 2470936840 mslabel: 9nKsWmxMvOQBYaz9xhRffWeWSUbCbnox6aQ4
a = ssrc: 2470936840 label: ce9235c5-f300-466a-aadd-b969dc2f3664
a = ssrc: 2969787502 cname: drYey7idt605CcEG
a = ssrc: 2969787502 msid: 9nKsWmxMvOQBYaz9xhRffWeWSUbCbnox6aQ4 ce9235c5-f300-466a-aadd-b969dc2f3664
a = ssrc: 2969787502 mslabel: 9nKsWmxMvOQBYaz9xhRffWeWSUbCbnox6aQ4
a = ssrc: 2969787502 label: ce9235c5-f300-466a-aadd-b969dc2f3664

For example, from this SDP-config it can be said that it is proposed to use H.264 and VP8 codecs for video and Opus for audio. In addition, you can get a lot of other useful information for phoning, such as priority of codecs, use of feedback, fir , nack , pli , profile and level for H.264 42e01f codec - Baseline 3.1, etc.

In the process of implementing video chat on the native WebRTC API, it is desirable to understand what SDP is, candidates (candidates), codecs, ICE, STUN, TURN and many other scary words.


WebRTC, Websockets and SIP


The concepts regarding WebRTC and Websocket are often confused. Sometimes SIP is added to this confusion.


It can be said for sure that WebRTC has no direct relation to either Websockets or SIP .

Websockets is only a convenient opportunity to transfer SDP from Boris to Alice. Instead of Websockets, we could use plain HTTP or send SDP text by mail. SDP messaging is signaling information and this data can be transmitted over any protocol you like. For browsers, the default data transfer protocols are: Websockets and HTTP. Therefore, Websockets is used as a more real-time option compared to HTTP. No audio or video is transmitted via Websockets. Only signaling information: text and commands.


SIP is a text messaging protocol. WebRTC is sometimes unfairly called SIP-ohm in the browser, most likely for the fact that SIP messages also use SDP for configuring codecs and establishing connections.

On the other hand, when we speak, for example, a SIP phone , we mean a device that, along with the SIP protocol (RFC3261), supports a dozen other network specifications and protocols: RTP, SDP, AVPF, etc.

Indeed, WebRTC at the network level uses similar building blocks with those that use a SIP phone (SRTP, STUN, etc.). Therefore, it can be said that both WebRTC and SIP devices / software use a similar technology base. But it would be incorrect to call WebRTC SIP in the browser, if only because there is no SIP in the out-of-box browsers.


WebRTC is a technology that has three main functions in terms of audio / video transmission:


And many support functions, such as jitter control, adaptive bitrate, network overload control, etc.

As described above, for the WebRTC media transfer to take place, Alice and Boris must exchange SDP, which contain detailed information about the formats of video streams, packetization and other parameters that determine exactly how the party will receive the video.

In addition to SDP exchange, a TURN server may be required, which will allow video traffic to pass through itself if the Peer-to-Peer connection is not established, for example, if Alice or Boris has a rather unfriendly (for example, symmetric ) NAT.

Suppose we wanted to add a third active participant or just a viewer to the chat. A good example in this case is the debate. Two participants are talking - the rest are watching. Another example is a chat for three or more participants.

When a third participant appears, the scheme is complicated by the fact that each of the participants has to capture and press already two video streams instead of one, and also establish connections in pairs , trying to overcome NAT . In this case, the connection setup time may dramatically increase and its stability decreases. Simultaneous compression and sending of two or more video streams creates a serious load on the processor and the network and affects the quality for the worse, especially on mobile devices:


Similar tasks such as


go beyond the successful applicability of peer-to-peer and require a central WebRTC server that manages these connections.


As mentioned above, there are services and servers, convenient and inconvenient APIs over the WebRTC API that allow you to speed up the development of video chats and work with more convenient abstractions, such as video stream (Stream), room (Room), publishing (Publisher), viewer (Subscriber) , etc.

For example, to create the simplest video chat it would be enough to exchange the names of the streams. Boris knows the flow of Alice. Alice knows the flow of Boris - and video chat is ready:


Sample video chat in the browser


In this article, we will show how the Streaming API works with Web Call Server 5 - a WebRTC server for video chatting and online broadcasting.

Video chat in action can be illustrated in the following two screenshots. The first Alice member will see a video chat like this:


The second Edward member will see a video chat like this:


In this example, the following happens:

  1. Alice sent a video stream from a browser named Alice to the server.
  2. Edward sent a video stream from a browser called Edward to the server.
  3. Alice took and lost the video stream named Edward.
  4. Edward took and lost the video stream with the name Alice.

As you can see from this example, we built a video chat based only on the fact that Alice and Edward know the names of each other's streams. We did not have to work directly with SDP, PeerConnection, NAT, TURN, etc.

Thus, video chat can be implemented simply by transferring the names of streams to those who have to lose them.

In this simple concept, you can use any front-end and back-end technologies, such as Jquery, Bootstrap, React, Angular, PHP, Java, .Net, etc. And the good news is that embedding support for video streams and video chat does not affect the existing web application. You control your video chat by simply allowing (or not allowing) its participants to play the desired video streams.

Video chat code in the browser


Now we will show how it looks in code. The video chat HTML page contains two main div elements :




You can give these elements any arbitrary identifiers, such as id = "captureVideo" or id = "playbackVideo", but it is important that these elements are present on the page.

A working HTML page with localVideo and remoteVideo blocks looks like this:

<html> <head> <script language="javascript" src="flashphoner.js"></script> <script language="javascript" src="video-chat.js"></script> </head> <body onLoad="init()"> <h1>Video Chat</h1> <div id="localVideo" style="width:320px;height:240px;border: 1px solid"></div> <div id="remoteVideo" style="width:320px;height:240px;border: 1px solid"></div> <input type="button" value="connect" onClick="connect()"/> <input type="button" value="publish" onClick="publish('Alice')"/> <input type="button" value="play" onClick="play('Edward')"/> <p id="status"></p> </body> </html>

Now we give the code that is directly responsible for sending and playing the video.

Send stream from webcam


When sending, we use the API method session.createStream (). Publish () and specify for this stream an HTML div element in which the localVideo video captured from the camera will be displayed, as well as the name of the Alice video stream, knowing that the other stream can play the video by the client.

 session.createStream({ name: "Alice", display: localVideo, cacheLocalResources: true, receiveVideo: false, receiveAudio: false }).on(Flashphoner.constants.STREAM_STATUS.PUBLISHING, function (publishStream) { setStatus(Flashphoner.constants.STREAM_STATUS.PUBLISHING); }).on(Flashphoner.constants.STREAM_STATUS.UNPUBLISHED, function () { setStatus(Flashphoner.constants.STREAM_STATUS.UNPUBLISHED); }).on(Flashphoner.constants.STREAM_STATUS.FAILED, function () { setStatus(Flashphoner.constants.STREAM_STATUS.FAILED); }).publish();

Play video from the server


During playback, we specify the name of the stream that will be played and the HTML div element remoteVideo , in which the stream received from the server will be played. The API method session.createStream (). Play () is used .

  session.createStream({ name: "Edward", display: remoteVideo, cacheLocalResources: true, receiveVideo: true, receiveAudio: true }).on(Flashphoner.constants.STREAM_STATUS.PLAYING, function (playStream) { setStatus(Flashphoner.constants.STREAM_STATUS.PLAYING); }).on(Flashphoner.constants.STREAM_STATUS.STOPPED, function () { setStatus(Flashphoner.constants.STREAM_STATUS.STOPPED); }).on(Flashphoner.constants.STREAM_STATUS.FAILED, function () { setStatus(Flashphoner.constants.STREAM_STATUS.FAILED); }).play();

While working with the server, the HTML page will receive various statuses, such as PLAYING , STOPPED for playback and PUBLISHING , UNPUBLISHED for publishing the stream.

Thus, the main thing that is required for video chatting is to place on the web page two div-blocks and connect the corresponding scripts that run stream.play () and stream.publish () by the stream name.

The full source code for the Two Way Streaming sample is available for download here .

Sample WebRTC video chat in an Android application


Video chat for Android will work exactly the same way as video chat in a browser. The application will establish a connection with the server, send a video stream from the camera of an Android device, receive and play another video stream from the server.

Below is the Android application Streaming Min (analogous to the Two Way Streaming example for video chat in the browser) that allows you to share video streams.


From the screenshot it can be seen that nothing has changed. We have two video windows. On the left is the video captured from the webcam, and on the right is the video that comes from the server. The exchange of video streams is carried out in the same way, using names. We publish one stream and lose the other.

Android video chat code


If to create a video chat for the browser, we used the Web SDK , which includes the flashphoner.js API script , to create a full-fledged native Android application, you need to import the Android SDK aar file into the project.

To figure out how this works, the easiest way is to build and run the Streaming Min sample based on the Android SDK. All examples are available in the github repository .

1. Download all the examples.

 git clone https://github.com/flashphoner/wcs-android-sdk-samples.git

2. Download the SDK

 wget https://flashphoner.com/downloads/builds/flashphoner_client/wcs-android-sdk/aar/wcs-android-sdk-1.0.1.25.aar

3. We hook the SDK as an aar file to the examples.

 cd export ./export.sh /tmp/wcs-android-sdk-1.0.1.25.aar

Please note that we have specified the export.sh script path to the downloaded file wcs-android-sdk-1.0.1.25.aar - Android SDK

As a result, in the export / output folder will be a fully configured project that can be opened in Android Studio

It remains only to collect examples using gradle .

1 - Create launch configuration


2 - Choose Gradle Script


3 - Run the build


As a result of the build, we need to get apk-files that can already be installed on the Android device.

In this example, we exchanged video streams with the browser. The test33 video stream was sent from the Android device to the server and played back in the browser. Video stream 8880 was sent by the browser and played on the Android device. In this way, we received two-way audio and video communication between the browser and the Android application.


If we used HTML div elements for video in the web version of video chat, in Android we use renderers

 private SurfaceViewRenderer localRender; private SurfaceViewRenderer remoteRender;


The localRenderer displays video captured from the camera of an Android device. The remoteRenderer displays the video that came from the server.

1. Create a connection to the server and indicate that renderers will be used ...

 sessionOptions = new SessionOptions(mWcsUrlView.getText().toString()); sessionOptions.setLocalRenderer(localRender); sessionOptions.setRemoteRenderer(remoteRender); ... session = Flashphoner.createSession(sessionOptions); … session.connect(new Connection());

2. Create a stream with an arbitrary name and publish the stream to the server.

 StreamOptions streamOptions = new StreamOptions(mPublishStreamView.getText().toString()); publishStream = session.createStream(streamOptions); ... publishStream.publish();

3. Specify the name of the stream during playback and take the stream from the server.

 StreamOptions streamOptions = new StreamOptions(mPlayStreamView.getText().toString()); playStream = session.createStream(streamOptions); ... playStream.play();

The full code for the class StreamingMinActivity.java is available here . And the code for the entire Streaming Min example for Android is available in the repository via this link .

Web call server


As a result, we showed how to organize a simple exchange of video streams between the HTML page in the browser and the Android application.

Video streams pass through the Web Call Server, which is simultaneously a signaling server and proxies through itself audio and video traffic.


Web Call Server is a server software that can be installed on a Linux system on a virtual server or on a physical (dedicated) server. WCS is a WebRTC video streaming server and can serve video streams from browsers, iOS and Android devices.

Links


Technologies and Protocols


WebRTC - WebRTC technology
SDP - Session description protocol, RFC
Websocket - Websocket Protocol, RFC

Server and API for developing video chat


Web Call Server - WebRTC video streaming server for video chat
Download Web Call Server - server installation
Web Call Server on EC2 - running a server image on Amazon EC2
Web SDK - Web SDK for developing video chat with WebRTC support
Android SDK - Android SDK for developing video chat with WebRTC support

Working examples


Web Two Way Streaming - an example of sharing video streams for the Web
Android Two Way Streaming - a sample video sharing application for Android

Source code examples


Web Two Way Streaming - code example video sharing for Web
Android Two Way Streaming - code example video sharing for Android

Source: https://habr.com/ru/post/324914/

More articles:


All Articles