WebRTC video chat development between iOS, Android and browser

In the previous article, we described the principles of video chat development between a browser and an Android device. Now let's try to complicate the task and implement the three-way video chat on the following platforms: Google Chrome on the desktop, Android application on the tablet, and iOS application for the Apple iPhone.

Recall the two basic principles of building video chat:

1. Each logged in user can send (publish) his video stream to the server.
2. Users know the names of each other's video streams and have the ability to play them.

Thus, in the video chat of three participants, each of the participants will have to play two video streams.


The sequence of nine actions is as shown below. First, the participants publish their video streams, then play each other's streams.
')
Publish streams

1. Alice: session.createStream({name:"stream_alice"}).publish(); 2. Boris: session.createStream({name:"stream_boris"}).publish(); 3. Anna: session.createStream({name:"stream_anna"}).publish(); 

Play streams

 4. Alice: session.createStream({name:"stream_anna"}).play(); 5. Alice: session.createStream({name:"stream_boris"}).play(); 6. Boris: session.createStream({name:"stream_alice"}).play(); 7. Boris session.createStream({name:"stream_anna"}).play(); 8. Anna: session.createStream({name:"stream_alice"}).play(); 9. Anna: session.createStream({name:"stream_boris"}).play(); 

The developer in this case is required to organize the transfer of the names and status of video streams to video chat participants. This can be done on any suitable technology, such as PHP, websockets, Node.js, etc., since the transfer of the name of the video stream is no different from the transfer of a plain text message from one user to another.

A stream can have three basic statuses: PUBLISHING , PLAYING , STOPPED

For successful playback, the requested stream must be in the active PUBLISHING status.


This diagram shows how it is possible to implement the exchange of names and status of streams, in a simplified version, when Alice shows her video stream to Boris and Anna. This procedure takes 8 steps , and can be called a signaling , because As a result of this procedure, coordination is carried out:

1. Alice sends a video stream to the WCS server.
2. Alice receives confirmation from the server in the form of PUBLISHING status
3. Alice sends Boris a message saying that her stream is ready for reproduction.
4. Alice sends Anna a message stating that her stream is ready to play.
5. Anna plays the video stream from the WCS server.
6. Anna receives confirmation of the status of PLAYING.
7. Boris plays the video stream from the WCS server.
8. Boris receives confirmation of the status of PLAYING.

As a result, an arbitrary interaction scenario can be implemented: with the connection of two or more users to the chat, with the connection of just the viewers, etc.

Rooms

Organizing the exchange of names of video streams and their status is not so difficult technically, but it requires some labor and work with the code.

On the other hand, for such a problem, some kind of universal solution suggests itself that could help users quickly reconcile the flows and get into the chat. This solution is called Room or Room API .

Indeed, if two or more users interact in the same context, then this is like a room. Inside the room, users see video streams of each other, they know who is in the room and can exchange messages, including private ones.

Thus, we have four objects that fully cover the work with the rooms:

• Room - room
• Stream - video stream
• Participant - participant
• Message - the message

The Room API allows you to cross-platform use the abstractions listed above: Room , Stream , Participant, and Message to implement the following functions:

Connection

• User can connect to the room.
• The user sets the name of the room when connecting.
• If a room with this name exists, the user enters this room.
• If there is no room with such a name, then a new room is created and the user enters the newly created room.
• The user receives notifications about the connections / disconnections of other participants.
• The user can get a list of participants.

Streaming

• The user can publish a video stream inside the room.
• The user receives notifications about the status of video streams of other participants.
• The user can play the video stream inside the room.

Messages

• Users can chat inside the room.
• Users can share images or other content if it is packaged in text format.
• A message can be sent to one or more participants.

Room API

For the Web platform, the rooms were implemented as a JavaScript module with the following main functions:

1. Get a connection to the server.

 var connection = Flashphoner.roomApi.connect({urlServer: "wss://host:8443", username: "Alice"}); 

2. Enter the room.

 connection.join({name: "room1"}); 

3. Get a list of room members.

 var participants = room.getParticipants(); 

4. Send your video stream to the room.

 room.publish({ display: document.getElementById("localDisplay"), constraints: constraints, record: false, receiveVideo: false, receiveAudio: false }); 

5. Reproduce the stream member.

 participant.getStreams()[0].play(document.getElementById(pDisplay)) 

6. We follow the participants of the room:

 connection.on(ROOM_EVENT.JOINED, function(participant){...}); connection.on(ROOM_EVENT.LEFT, function(participant){...}); connection.on(ROOM_EVENT.PUBLISHED, function(participant){...}); 

JOINED - a new member joined the room.
LEFT - member out
PUBLISHED - Member has posted its video stream.

7. Receive messages from other participants.

 connection.on(ROOM_EVENT.MESSAGE, function(participant){...}); 

8. We send a private message to a specific participant.

 participants[i].sendMessage(message); 

9. Send a message to all participants immediately.

 var participants = room.getParticipants(); for (var i = 0; i < participants.length; i++) { participants[i].sendMessage(message); } } 

Thus, the implementation of the rooms gives a simple exchange of messages and the status of all participants connected to the room.

Room API Restrictions

All the logic when working with rooms falls on the client. The server controls only the basic functionality of the room:

• notifications about connecting / disconnecting users in the room
• notifications about creating / disabling video streams by users inside the room
• message routing to specified members

Thus, authentication, access rights, the roles of the participants (moderator, viewer, presenter), and other logic should be implemented on the client and / or third-party back end. The rooms only help to quickly exchange information about video streams and statuses.

Rooms for Web, Android, iOS

In each of the SDK (Web, Android, iOS) to work with the server, there is an API for working with rooms.
Examples of entering the room:

Web SDK

 connection.join({name: "room1"}); 

Android SDK

 Room room = roomManager.join(roomOptions); 

iOS SDK

 room = [roomManager join:options]; 

Thus, three participants from three different platforms can enter the same room:

• Web browser
• Mobile application for Android
• IOS Mobile App

Test applications for working with rooms

Below we give three test applications for working with rooms. Each of them is open source and can be compiled from sources.

Each of the following applications allows you to exchange video streams and messages to three participants:

• Conference for Web
• Conference for Android
• Conference for iOS

Conference for Web

The code for this application is available for download here .

In the HTML code you can find two div elements.

 <div id="participant1Display" class="display"></div> 

 <div id="participant2Display" class="display"></div> 

These items are responsible for displaying video of participants.

The div element localDisplay is responsible for displaying the video captured from the camera.

 <div id="localDisplay" class="fp-localVideo"></div> 

Using the Join / Leave buttons you can enter and exit the room, respectively. Using Stop / Start, you can send a video to a room or stop broadcasting. Login field must be unique, because identifies the participant.


The next interface element is text chat. This chat displays messages received or sent by other users, and also displays information about events occurring in the room.


And the last element is the link to the invitation. If the user is logged in first, a new room is created. In this case, named roomName = room-fb41b7. If the second user specifies the same roomName, he will fall into the same room. In the Conference for Web application, invitations are implemented by generating an input link containing a roomName as a parameter. In versions of the application for Android and iOS, the name of the room is indicated directly in the interface.


Thus, the Conference for Web test application implements all the functions of the rooms incorporated above and allows several users to connect to the same room and exchange video streams and text messages.

The screenshot below shows how the room with three participants works. Three tabs of the Google Chrome browser were opened and a connection to the room was initiated on each tab.

Conference for Android

The code for this application is available for download here .

In the application, you can enter the name of the room directly from the UI. The rest of the application is very similar to Conference for Web , which was described above and has the same interface elements:

• Leave / Join buttons for entering and exiting the room.
• Two video windows for playing video participants.
• One video window to display the capture video from the camera.
• Text chat with system information and member messages

Conference for iOS

The application code is available for download here .

The interface of the application for iOS is practically no different from the application for Android, up to the name of the buttons and controls.

As a result, we did a test and collected all three platforms in the same room with the number 3119d6.

The Google Chrome browser is a hare that is selected from the hole:


Android 5.1.1 on the Asus tablet is a flower pot.


iOS 10.1.1 on iPhone 6 is Ficus Benjamin on the windowsill.


Below is a full screenshot of the iOS device:


Thus, we have completed testing and review of all three applications built on the basis of the Room API, and can proceed to the source code and build.

Building the Conference for iOS application from source

In the previous article, we showed how to build an example of video chat for android and explain how the video chat code is arranged in the browser.

Here we show how to build the Conference for iOS example and describe the main pieces of the example code.

The first thing you need to get Mac-hardware and install the latest Xcode .

The build procedure requires installing Cocoapods , downloading the sample code and the SDK for iOS. We will collect in the terminal, and then we will open the project in Xcode.

1. Install Cocoapods

 sudo gem install cocoapods 

2. Download all the examples from github

 git clone https://github.com/flashphoner/wcs-ios-sdk-samples.git 

3. Download iOS SDK

 wget https://flashphoner.com/downloads/builds/flashphoner_client/wcs-ios-sdk/WCS-iOS-SDK-2.3.0.tar.gz 

4. Unpack the archive

 tar -xvzf WCS-iOS-SDK-2.3.0.tar.gz 

5. Copy the folder FPWCSApi2.framework into examples

 cp -R FPWCSApi2.framework wcs-ios-sdk-samples 

6. Run the assembly.

 ./build_example.sh 

If the build is successful, ** ARCHIVE SUCCEDED ** is output in the terminal.


After the examples are collected from the console using Cocoapods , all dependencies are pulled and configured and further examples can be built directly from Xcode .

7. Open WCSExample.xcworkspace in Xcode


8. Select Generic iOS Device for assembly purposes. Launch the assembly of the Conference example from the Product / Build menu and wait for completion.


9. Connect the iPhone via USB and run the assembled Conference application. Debug information appears in the console.


10. The Conference application appears on the iPhone.


Thus, we compiled the Conference mobile app for iOS from source and used the iOS SDK version 2.3.0 + Cocoapods for this build. As a result, we were able to run this application on an iPhone 6 connected via USB to the Mac on which this assembly was made.

Code for Web, Android, iOS video chat

Above, we gave examples of applications for three platforms that use the Room API and exchange video streams inside the created room. We will try to briefly list the main pieces of code responsible for the work of the video chat in each of these three platforms:

	Web	Android	iOS
Code	Javascript	Java	Objective c
Main code	conference.js	ConferenceActivity.java	ViewController.m
Connect to server	connection = Flashphoner.roomApi.connect({urlServer: url, username: username});	roomManager = Flashphoner.createRoomManager(roomManagerOptions);	roomManager = [FPWCSApi2 createRoomManager:options error:&error];
Connection to the room	connection.join({name: getRoomName()});	room = roomManager.join(roomOptions);	room = [roomManager join:options];
Getting the list of participants	var participants = room.getParticipants();	room.getParticipants()	NSDictionary *participants = [room getParticipants];
Sending a video stream to the room	room.publish({ display: document.getElementById("localDisplay"), constraints: constraints, record: false, receiveVideo: false, receiveAudio: false });	stream = room.publish(localRenderer);	publishStream = [room publish:_localDisplay];
Play the video stream of the participant	participant.getStreams()[0].play(document.getElementById(pDisplay));	participant.play(participantView.surfaceViewRenderer);	[participant play:pv.display];
We track the accession of new members to the room	connection.on(ROOM_EVENT.JOINED, function(participant){...});	public void onJoined(final Participant participant){...};	[room on:kFPWCSRoomParticipantEventJoined participantCallback:^(FPWCSApi2Room *room, FPWCSApi2RoomParticipant *participant){...}]
We trace members leaving the room	connection.on(ROOM_EVENT.LEFT, function(participant){...});	public void onLeft(final Participant participant){...}	[room on:kFPWCSRoomParticipantEventLeft participantCallback:^(FPWCSApi2Room *room, FPWCSApi2RoomParticipant *participant){...}]
We trace the publication of a video stream by the participant of the room	connection.on(ROOM_EVENT.PUBLISHED, function(participant){...});	public void onPublished(final Participant participant){...}	[room on:kFPWCSRoomParticipantEventPublished participantCallback:^(FPWCSApi2Room *room, FPWCSApi2RoomParticipant *participant){...}]
We receive an incoming message from other participants	connection.on(ROOM_EVENT.MESSAGE, function(participant){...});	public void onMessage(final Message message){...}	[room onMessageCallback:^(FPWCSApi2Room *room, FPWCSApi2RoomMessage *message){...}]
Send a message to one of the participants	participants[i].sendMessage(message);	participant.sendMessage(text);	[participant sendMessage:_messageBody.text];

Total we get 10 basic designs for each of the platforms. Each of the listed constructions can be found in the corresponding file with the code.

All three Web, iOS and Android applications were tested with the latest build of Web Call Server 5 - video chat servers and low-latency broadcasts with support for WebRTC technology.

Links

Web Call Server - server for WebRTC video chat
EC2 launch - launch a finished image on Amazon EC2
Install - install server on VPS or Dedicated host
Cocoapods - dependency manager for build

	Web SDK	Android SDK	iOS SDK
Sdk	html	html	html
Download SDK	tar.gz	aar	tar.gz
Sample Chat Room Application	WCS demo	Google play	Ad hoc only
The main sample code file	Conference.js	ConferenceActivity.java	ViewController.m
Source code of all examples	github	github	github
Description of the procedure for assembling examples from source code	html	html	html
Description of the code of the Conference rooms example	html	html