Geekly Articles each Day
📜 ⬆️ ⬇️

We pull video from YouTube and distribute via WebRTC in real-time



The task is as follows. To share video with YouTube in real time by several users. Viewers should receive video at the same time, with minimal delay.

Movie Stream


It is clear that if each of the spectators just starts playing the video, the goal will not be achieved, because one will get the video faster, the other slower. There will be an uncontrollable scatter.
')
In order to scatter was not necessary to distribute this video all at once. This can be realized if you wrap the clip in Live-stream. We will show how to do this using the bundle of this library with ffmpeg.


We need to implement the scheme described above. Namely, ydl connects to YouTube and starts downloading the video. FFmpeg picks up the downloaded video, wraps it in an RTMP stream and sends it to the server. The server distributes the received stream as WebRTC in real time.

Installing youtube-dl


We start by installing youtube-dl. The installation process on Linux is extremely simple and is described in detail in the Readme under Lin and under Win .

1. Download.

curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl

2. We give launch rights

 chmod a+rx /usr/local/bin/youtube-dl

That's all. YouTube downloader is ready to go.

Take a YouTube video and look at its meta data:

 youtube-dl --list-formats https://www.youtube.com/watch?v=9cQT4urTlXM

The result will be:

 [youtube] 9cQT4urTlXM: Downloading webpage [youtube] 9cQT4urTlXM: Downloading video info webpage [youtube] 9cQT4urTlXM: Extracting video information [youtube] 9cQT4urTlXM: Downloading MPD manifest [info] Available formats for 9cQT4urTlXM: format code extension resolution note 171 webm audio only DASH audio 8k , vorbis@128k, 540.24KiB 249 webm audio only DASH audio 10k , opus @ 50k, 797.30KiB 250 webm audio only DASH audio 10k , opus @ 70k, 797.30KiB 251 webm audio only DASH audio 10k , opus @160k, 797.30KiB 139 m4a audio only DASH audio 53k , m4a_dash container, mp4a.40.5@ 48k (22050Hz), 10.36MiB 140 m4a audio only DASH audio 137k , m4a_dash container, mp4a.40.2@128k (44100Hz), 27.56MiB 278 webm 256x144 144p 41k , webm container, vp9, 30fps, video only, 6.54MiB 242 webm 426x240 240p 70k , vp9, 30fps, video only, 13.42MiB 243 webm 640x360 360p 101k , vp9, 30fps, video only, 20.55MiB 160 mp4 256x144 DASH video 123k , avc1.4d400c, 15fps, video only, 24.83MiB 134 mp4 640x360 DASH video 138k , avc1.4d401e, 30fps, video only, 28.07MiB 244 webm 854x480 480p 149k , vp9, 30fps, video only, 30.55MiB 135 mp4 854x480 DASH video 209k , avc1.4d401f, 30fps, video only, 42.42MiB 133 mp4 426x240 DASH video 274k , avc1.4d4015, 30fps, video only, 57.63MiB 247 webm 1280x720 720p 298k , vp9, 30fps, video only, 59.25MiB 136 mp4 1280x720 DASH video 307k , avc1.4d401f, 30fps, video only, 62.58MiB 17 3gp 176x144 small , mp4v.20.3, mp4a.40.2@ 24k 36 3gp 320x180 small , mp4v.20.3, mp4a.40.2 43 webm 640x360 medium , vp8.0, vorbis@128k 18 mp4 640x360 medium , avc1.42001E, mp4a.40.2@ 96k 22 mp4 1280x720 hd720 , avc1.64001F, mp4a.40.2@192k (best)


Install ffmpeg


Next, set the ffmpeg standard spells:

 wget http://ffmpeg.org/releases/ffmpeg-3.3.4.tar.bz2 tar -xvjf ffmpeg-3.3.4.tar.bz2 cd ffmpeg-3.3.4 ./configure --enable-shared --disable-logging --enable-gpl --enable-pthreads --enable-libx264 --enable-librtmp make make install

Checking what happened

 ffmpeg -v

Now the fun part. Library youtube-dl is designed for download. It is called YouTube Download. Those. You can download the youtube video completely and after that clip it through ffmpeg as a file.

But imagine such a user. Sit in a web conference marketer, manager and programmer. Marketer wants to show everyone in real-time video from YouTube, which weighs, say 300 megabytes. Agree, there will be a certain awkwardness, if you need to deflate the entire movie before you start showing it.

  1. Marketer says - "Now, colleagues, let's see this video with cats, it fully meets our strategy of entering the market," and presses the button "show everyone the video."
  2. A preloader appears on the screen: “Wait, the video with the cats is downloading. It takes less than 10 minutes. ”
  3. The manager goes to drink coffee, and the programmer - to read habr.

In order not to make people wait, you need realtime. It is necessary to pick up the video directly during the download, on the fly to wrap the stream and distribute in real time. Next we show how to do it.

Data transfer from youtube-dl to ffmpeg


Grabber youtube-dl saves the stream in the file system. You need to connect to this stream and organize a read from the ffmpeg file as you download it using youtube-dl.

To combine these two processes: downloading and streaming ffmpeg, we need a small binding script.

 #!/usr/bin/python import subprocess import sys def show_help(): print 'Usage: ' print './streamer.py url streamName destination' print './streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM streamName rtmp://192.168.88.59:1935/live' return def streamer() : url = sys.argv[1] if not url : print 'Error: url is empty' return stream_id = sys.argv[2] if not stream_id: print 'Error: stream name is empty' return destination = sys.argv[3] if not destination: print 'Error: destination is empty' return _youtube_process = subprocess.Popen(('youtube-dl','-f','','--prefer-ffmpeg', '--no-color', '--no-cache-dir', '--no-progress','-o', '-', '-f', '22/18', url, '--reject-title', stream_id),stdout=subprocess.PIPE) _ffmpeg_process = subprocess.Popen(('ffmpeg','-re','-i', '-','-preset', 'ultrafast','-vcodec', 'copy', '-acodec', 'copy','-threads','1', '-f', 'flv',destination + "/" + stream_id), stdin=_youtube_process.stdout) return if len(sys.argv) < 4: show_help() else: streamer()

This python script does the following:

  1. Creates a _youtube_process subprocess of reading a movie by youtube-dl library
  2. Creates a second _ffmpeg_process subprocess to which data from the first is passed through a pipe. This process already creates an RTMP stream and sends it to the server at the specified address.


Script Testing


To run the script you need to install python. Download here .

We used version 2.6.6 for testing. Most likely any version is suitable, because The script is quite simple and its task is to transfer from one process to another.

Running the script:
 python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://192.168.88.59:1935/live

As you can see, three arguments are passed:

  1. Address youtube video.
    www.youtube.com/watch?v=9cQT4urTlXM
  2. The name of the stream with which the RTMP broadcast will take place.
    stream1
  3. The address of the RTMP server.
    rtmp: //192.168.88.59: 1935 / live

For testing, we will use the Web Call Server . He is able to receive RTMP streams and distribute them via WebRTC. Here you can download and install WCS5 on your VPS or local test server running Linux.

Testing scheme with Web Call Server:


Below we use one of the demo servers for the test:

 rtmp://wcs5-eu.flashphoner.com:1935/live

This is the RTMP address that you need to transfer to the streamer.py script in order to quickly test the broadcast with our demo server.

The launch should look like this:

 python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://wcs5-eu.flashphoner.com:1935/live

In the stdout console, we see the following output:

 # python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://wcs5-eu.flashphoner.com:1935/live ffmpeg version 3.2.3 Copyright (c) 2000-2017 the FFmpeg developers built with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-11) configuration: --enable-shared --disable-logging --enable-gpl --enable-pthreads --enable-libx264 --enable-librtmp --disable-yasm libavutil 55. 34.101 / 55. 34.101 libavcodec 57. 64.101 / 57. 64.101 libavformat 57. 56.101 / 57. 56.101 libavdevice 57. 1.100 / 57. 1.100 libavfilter 6. 65.100 / 6. 65.100 libswscale 4. 2.100 / 4. 2.100 libswresample 2. 3.100 / 2. 3.100 libpostproc 54. 1.100 / 54. 1.100 ]# [youtube] 9cQT4urTlXM: Downloading webpage [youtube] 9cQT4urTlXM: Downloading video info webpage [youtube] 9cQT4urTlXM: Extracting video information [youtube] 9cQT4urTlXM: Downloading MPD manifest [download] Destination: - Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: isommp42 creation_time : 2016-08-23T12:21:06.000000Z Duration: 00:29:59.99, start: 0.000000, bitrate: N/A Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 288 kb/s, 30 fps, 30 tbr, 90k tbn, 60 tbc (default) Metadata: creation_time : 2016-08-23T12:21:06.000000Z handler_name : ISO Media file produced by Google Inc. Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default) Metadata: creation_time : 2016-08-23T12:21:06.000000Z handler_name : ISO Media file produced by Google Inc. Output #0, flv, to 'rtmp://192.168.88.59:1935/live/stream1': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: isommp42 encoder : Lavf57.56.101 Stream #0:0(und): Video: h264 (Main) ([7][0][0][0] / 0x0007), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 288 kb/s, 30 fps, 30 tbr, 1k tbn, 90k tbc (default) Metadata: creation_time : 2016-08-23T12:21:06.000000Z handler_name : ISO Media file produced by Google Inc. Stream #0:1(und): Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, 125 kb/s (default) Metadata: creation_time : 2016-08-23T12:21:06.000000Z handler_name : ISO Media file produced by Google Inc. Stream mapping: Stream #0:0 -> #0:0 (copy) Stream #0:1 -> #0:1 (copy) frame= 383 fps= 30 q=-1.0 size= 654kB time=00:00:12.70 bitrate= 421.8kbits/s speed= 1x

If you quickly run through this log, you can understand that the following is happening:

  1. A video page opens.
  2. Retrieves data about video formats.
  3. Downloadable mp4 video 1280x720, H.264 + AAC
  4. Runs ffmpeg, picks up the downloaded data and stream over RTMP with a bit rate of 421 kbps. Such a meager bit rate is due to the selected roller with a timer. A normal video will give an order of magnitude greater value of the bitrate.

After the streaming process starts, we try to play the stream in the WebRTC player . The stream name is set in the Stream field, and the server address in the Server field. Connection to the server occurs via the Websocket (wss) protocol, and the stream comes to the player via WebRTC (UDP).


We specifically took this particular video on YouTube to be able to test the real-timeness of the stream, because our ultimate goal was to deliver the stream from YouTube to all viewers at the same time, with minimal delay and time variation. Movie with a millisecond timer , could not be more suitable for such a test.

The test itself is very simple. We open two browser tabs (we simulate two viewers), we play this stream with a timer according to our scheme, and we take several screenshots to capture the difference in the time of arrival of the video. Next, compare the milliseconds and see who got the video before, and who later and how much.

We get the following results:

Test 1


Test 2


Test 3


As you can see, each of the viewers sees the same video, with a spread of not more than 130 milliseconds.

Thus, the task of real-time broadcast video from YouTube to WebRTC is solved. Viewers received the stream almost simultaneously. The manager did not leave to drink coffee, the programmer didn’t read Habr, and the marketer successfully showed everyone a video with cats.

Good streaming!

Links


youtube-dl - library for downloading videos from YouTube
ffmpeg - RTMP encoder
Web Call Server - a server that can split RTMP stream over WebRTC
streamer.py - a script for integrating youtube-dl and ffmpeg with sending an RTMP stream

Source: https://habr.com/ru/post/338098/

More articles:


All Articles