How I made (almost) useless Javascript webcam streaming
In this article, I want to share my attempts to stream video via websockets without using third-party browser plugins such as Adobe Flash Player. What came of this read on.
Adobe Flash - formerly Macromedia Flash, is a platform for creating applications that run in a web browser. Prior to the implementation of the Media Stream API, it was almost the only platform for streaming video and voice from a webcam, as well as for creating various conferences and chats in a browser. The protocol for transmitting media information RTMP (Real Time Messaging Protocol) was actually closed for a long time, which meant: if you want to raise your streaming service, please use the software from Adobe itself - Adobe Media Server (AMS).
After some time, in 2012, Adobe “surrendered and spat out” the specification of the RTMP protocol, which contained errors and, in fact, was not complete. By that time, developers began to make their implementations of this protocol, so the Wowza server appeared. In 2011, Adobe filed a lawsuit against Wowza for the illegal use of patents related to RTMP, after 4 years the conflict was resolved by the world.
The Adobe Flash platform has been around for over 20 years, during this time many critical vulnerabilities were discovered, support was announced to be stopped by 2020, so there are not so many alternatives for the streaming service.
')
For my project, I immediately decided to completely abandon the use of Flash in the browser. The main reason I indicated above, also Flash is not supported at all on mobile platforms, and I really did not want to deploy Adobe Flash for development on windows (wine emulator). So I started writing a client in JavaScript. It will be just a prototype, since later on I learned that streaming can be done much more efficiently based on p2p, only I will have peer - server - peers, but more about that another time, because it is not ready yet.
To get started, we need the webscokets server itself. I made the simplest one based on the melody go package:
On the client (broadcasting side), you must first access the camera. This is done through the MediaStream API .
We get access (resolution) to the camera / microphone through the Media Devices API . This API provides the MediaDevices.getUserMedia () method, which shows a popup. a window asking the user for permission to access the camera and / or microphone. I would like to note that I conducted all the experiments in Google Chrome, but I think that in Firefox everything will work in approximately the same way.
Next, getUserMedia () returns a Promise, into which a MediaStream object is passed - a stream of video and audio data. We assign this object to the video element property in src. Code:
To broadcast a video stream through sockets, it is necessary to somehow encode it somewhere, buffer it, and transmit it in parts. The raw video stream cannot be transmitted through websockets. This is where the MediaRecorder API comes in . This API allows you to encode and break a stream into pieces. I do coding to compress the video stream, so that I can drive less bytes over the network. Having broken into pieces, it is possible to send each piece to websocket. Code:
Now add the transfer on websockets. Surprisingly, this requires only a WebSocket object. It has only two send and close methods. The names speak for themselves. Extended code:
The broadcasting side is ready! Now let's try to take a video stream and show it on the client. What do we need for this? First of all, of course, a socket connection. We hang a “listener” on the WebSocket object, subscribe to the 'message' event. Having received a piece of binary data, our server broadcasts it to subscribers, that is, clients. At the same time, the callback function connected with the “listener” of the 'message' event is triggered on the client, the object itself is passed into the function argument - a piece of the video stream encoded by vp8.
For a long time I was trying to understand why it is impossible to immediately send the received pieces to the video element for playback, but it turned out to be impossible, of course, you first need to put the piece in a special buffer attached to the video element, and only then it starts playing the video stream. To do this, you need the MediaSource API and the FileReader API .
MediaSource acts as a kind of intermediary between the media playback object and the source of this media stream. The MediaSource object contains a pluggable buffer for the video / audio stream source. One feature is that the buffer can only contain Uint8 data, so FileReader is required to create such a buffer. Take a look at the code and it will become more clear:
The prototype of the streaming service is ready. The main disadvantage is that video playback will be 100 ms behind the transmitting side, we set this ourselves when splitting the video stream before sending it to the server. Moreover, when I checked on my laptop, I gradually accumulated a lag between the transmitting and receiving sides, this was clearly visible. I started looking for ways to overcome this shortcoming, and ... came across the RTCPeerConnection API , which allows you to transfer a video stream without tricks like splitting a stream into pieces. The accumulated lag, I think, is due to the fact that in the browser before the transfer, each piece is transcoded to the webm format. I didn’t dig any further, but began to study WebRTC, I think about the results of my research, I’ll write a separate article if I find this community interesting.
Adobe Flash - formerly Macromedia Flash, is a platform for creating applications that run in a web browser. Prior to the implementation of the Media Stream API, it was almost the only platform for streaming video and voice from a webcam, as well as for creating various conferences and chats in a browser. The protocol for transmitting media information RTMP (Real Time Messaging Protocol) was actually closed for a long time, which meant: if you want to raise your streaming service, please use the software from Adobe itself - Adobe Media Server (AMS).
After some time, in 2012, Adobe “surrendered and spat out” the specification of the RTMP protocol, which contained errors and, in fact, was not complete. By that time, developers began to make their implementations of this protocol, so the Wowza server appeared. In 2011, Adobe filed a lawsuit against Wowza for the illegal use of patents related to RTMP, after 4 years the conflict was resolved by the world.
The Adobe Flash platform has been around for over 20 years, during this time many critical vulnerabilities were discovered, support was announced to be stopped by 2020, so there are not so many alternatives for the streaming service.
')
For my project, I immediately decided to completely abandon the use of Flash in the browser. The main reason I indicated above, also Flash is not supported at all on mobile platforms, and I really did not want to deploy Adobe Flash for development on windows (wine emulator). So I started writing a client in JavaScript. It will be just a prototype, since later on I learned that streaming can be done much more efficiently based on p2p, only I will have peer - server - peers, but more about that another time, because it is not ready yet.
To get started, we need the webscokets server itself. I made the simplest one based on the melody go package:
Server Code
package main import ( "errors" "github.com/go-chi/chi" "gopkg.in/olahol/melody.v1" "log" "net/http" "time" ) func main() { r := chi.NewRouter() m := melody.New() m.Config.MaxMessageSize = 204800 r.Get("/", func(w http.ResponseWriter, r *http.Request) { http.ServeFile(w, r, "public/index.html") }) r.Get("/ws", func(w http.ResponseWriter, r *http.Request) { m.HandleRequest(w, r) }) // m.HandleMessageBinary(func(s *melody.Session, msg []byte) { m.BroadcastBinary(msg) }) log.Println("Starting server...") http.ListenAndServe(":3000", r) } On the client (broadcasting side), you must first access the camera. This is done through the MediaStream API .
We get access (resolution) to the camera / microphone through the Media Devices API . This API provides the MediaDevices.getUserMedia () method, which shows a popup. a window asking the user for permission to access the camera and / or microphone. I would like to note that I conducted all the experiments in Google Chrome, but I think that in Firefox everything will work in approximately the same way.
Next, getUserMedia () returns a Promise, into which a MediaStream object is passed - a stream of video and audio data. We assign this object to the video element property in src. Code:
Broadcast side
<style> #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; } </style> </head> <body> <!-- "" --> <video autoplay id="videoObjectHtml5ApiServer"></video> <script type="application/javascript"> var video = document.getElementById('videoObjectHtml5ApiServer'); // MediaDevices API, ( ) // getUserMedia , video if (navigator.mediaDevices.getUserMedia) { navigator.mediaDevices.getUserMedia({video: true}).then(function (stream) { // video , video.srcObject = s; }); } </script> To broadcast a video stream through sockets, it is necessary to somehow encode it somewhere, buffer it, and transmit it in parts. The raw video stream cannot be transmitted through websockets. This is where the MediaRecorder API comes in . This API allows you to encode and break a stream into pieces. I do coding to compress the video stream, so that I can drive less bytes over the network. Having broken into pieces, it is possible to send each piece to websocket. Code:
We encode the video stream, beat it to pieces
<style> #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; } </style> </head> <body> <!-- "" --> <video autoplay id="videoObjectHtml5ApiServer"></video> <script type="application/javascript"> var video = document.getElementById('videoObjectHtml5ApiServer'); // MediaDevices API, ( ) // getUserMedia , video if (navigator.mediaDevices.getUserMedia) { navigator.mediaDevices.getUserMedia({video: true}).then(function (stream) { // video , video.srcObject = s; var recorderOptions = { mimeType: 'video/webm; codecs=vp8' // webm vp8 }, mediaRecorder = new MediaRecorder(s, recorderOptions ); // MediaRecorder mediaRecorder.ondataavailable = function(e) { if (e.data && e.data.size > 0) { // e.data } } mediaRecorder.start(100); // 100 }); } </script> Now add the transfer on websockets. Surprisingly, this requires only a WebSocket object. It has only two send and close methods. The names speak for themselves. Extended code:
We transmit the video stream to the server
<style> #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; } </style> </head> <body> <!-- "" --> <video autoplay id="videoObjectHtml5ApiServer"></video> <script type="application/javascript"> var video = document.getElementById('videoObjectHtml5ApiServer'); // MediaDevices API, ( ) // getUserMedia , video if (navigator.mediaDevices.getUserMedia) { navigator.mediaDevices.getUserMedia({video: true}).then(function (stream) { // video , video.srcObject = s; var recorderOptions = { mimeType: 'video/webm; codecs=vp8' // webm vp8 }, mediaRecorder = new MediaRecorder(s, recorderOptions ), // MediaRecorder socket = new WebSocket('ws://127.0.0.1:3000/ws'); mediaRecorder.ondataavailable = function(e) { if (e.data && e.data.size > 0) { // e.data socket.send(e.data); } } mediaRecorder.start(100); // 100 }).catch(function (err) { console.log(err); }); } </script> The broadcasting side is ready! Now let's try to take a video stream and show it on the client. What do we need for this? First of all, of course, a socket connection. We hang a “listener” on the WebSocket object, subscribe to the 'message' event. Having received a piece of binary data, our server broadcasts it to subscribers, that is, clients. At the same time, the callback function connected with the “listener” of the 'message' event is triggered on the client, the object itself is passed into the function argument - a piece of the video stream encoded by vp8.
We accept a video stream
<style> #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; } </style> </head> <body> <!-- "" --> <video autoplay id="videoObjectHtml5ApiServer"></video> <script type="application/javascript"> var video = document.getElementById('videoObjectHtml5ApiServer'), socket = new WebSocket('ws://127.0.0.1:3000/ws'), arrayOfBlobs = []; socket.addEventListener('message', function (event) { // "" arrayOfBlobs.push(event.data); // readChunk(); }); </script> For a long time I was trying to understand why it is impossible to immediately send the received pieces to the video element for playback, but it turned out to be impossible, of course, you first need to put the piece in a special buffer attached to the video element, and only then it starts playing the video stream. To do this, you need the MediaSource API and the FileReader API .
MediaSource acts as a kind of intermediary between the media playback object and the source of this media stream. The MediaSource object contains a pluggable buffer for the video / audio stream source. One feature is that the buffer can only contain Uint8 data, so FileReader is required to create such a buffer. Take a look at the code and it will become more clear:
Play the video stream
<style> #videoObjectHtml5ApiServer { width: 320px; height: 240px; background: #666; } </style> </head> <body> <!-- "" --> <video autoplay id="videoObjectHtml5ApiServer"></video> <script type="application/javascript"> var video = document.getElementById('videoObjectHtml5ApiServer'), socket = new WebSocket('ws://127.0.0.1:3000/ws'), mediaSource = new MediaSource(), // MediaSource vid2url = URL.createObjectURL(mediaSource), // URL arrayOfBlobs = [], sourceBuffer = null; // , - socket.addEventListener('message', function (event) { // "" arrayOfBlobs.push(event.data); // readChunk(); }); // MediaSource , // / // , , // , mediaSource.addEventListener('sourceopen', function() { var mediaSource = this; sourceBuffer = mediaSource.addSourceBuffer("video/webm; codecs=\"vp8\""); }); function readChunk() { var reader = new FileReader(); reader.onload = function(e) { // FileReader , // "" Uint8Array ( Blob) , // , / sourceBuffer.appendBuffer(new Uint8Array(e.target.result)); reader.onload = null; } reader.readAsArrayBuffer(arrayOfBlobs.shift()); } </script> The prototype of the streaming service is ready. The main disadvantage is that video playback will be 100 ms behind the transmitting side, we set this ourselves when splitting the video stream before sending it to the server. Moreover, when I checked on my laptop, I gradually accumulated a lag between the transmitting and receiving sides, this was clearly visible. I started looking for ways to overcome this shortcoming, and ... came across the RTCPeerConnection API , which allows you to transfer a video stream without tricks like splitting a stream into pieces. The accumulated lag, I think, is due to the fact that in the browser before the transfer, each piece is transcoded to the webm format. I didn’t dig any further, but began to study WebRTC, I think about the results of my research, I’ll write a separate article if I find this community interesting.
Source: https://habr.com/ru/post/461769/
All Articles