We force FFMPEG to change HLS flows depending on the current throughput
Hello, residents of Habr. Today I want to tell a story about how I had to dive into the depths of ffmpeg without preparation. This article will be a guide for those who need the ability of FFMPEG to work correctly with HLS streams (namely, a change of flows depending on the current network bandwidth).
Let's start a little with background. Not so long ago, we had a project, android tv, in which one of the features was playing several videos at the same time, that is, the user looks at the screen and sees 4 videos. Then he chooses one of them and watches it already in full screen. The task is clear, it remains only to do. The peculiarity is that the video comes in the HLS format. I think that if you are reading this, you are already familiar with HLS, but still briefly - we are given a file, in which there are links to several streams that should change depending on the current Internet speed.
The first thing we started to implement this feature is the EXOPlayer cherer. Which was quite logical, since EXOPlayer uses hardware codecs to play the video stream. But it turned out that EXO has its dark side. When using EXO to run more than one thread, no one knows what will happen. In our case, when we launched 4 threads, on some devices everything worked well, on some only 3 worked, and the fourth one did not start, and on some, for example, something else happened on Nexus 7 2013. When Nexus 72013 ran more than 1 stream, the hardware codecs just crashed and no videos worked, not only in our application, but also in other applications that use hardware codecs. The only way to raise them is to reboot the device. As it turned out, this topic was devoted to the githaba topic . As it became clear, we cannot use hardware codecs, which means that we need to use software codecs and I remind you that the main task was to play 4 videos at the same time.
And the search began and we searched for a long time and we tried a lot, but the only thing that suited us was IJKPlayer . This is a player that is a ffmpeg wrapper. He played HLS, played them in 4 streams, and also played other streams that EXOplayer did not play on all devices (for example, HEVC). And for a very long time everything was fine, until we began to notice that the player always plays the same stream and does not change it depending on the network bandwidth. For the small videos, the preview was not a problem, but for full screen it was a problem.
After searching, it turned out that the streams do not change, and the host IJKPplayer advised to parse the streams separately from the player and launch the one that is needed (just like the ffmpeg ticket). Naturally, this did not fit because the player must adapt itself to the Internet. The problem is a problem, but it must be solved. There was nothing on the Internet to find so that it was decided to personally add logic to changing the flow to lib. But before you do something, you need to understand where to do it. FFMPEG itself is a very big way and it’s not so easy to understand what is what, but I outlined for you a few basic places that we will need to work with.
')
So, the main points that we need to know:
So, summarize our tasks:
Listing:
Now go to the listing of the ffmpeg
Add to avio.c
In hls.c, the read_data method will look like this.
Remaining trifles we add new files to the makefile inside libavformat in HEADERS and OBJS we add the corresponding references
We also add the IjkMediaPlayer_freeBitateWorkData method to ijkplayer_jni.c, which we will call after the view is complete, to clear the data.
Everything, our implementation is ready, now it remains to rebuild and watch videos with changing stream.
Let's start a little with background. Not so long ago, we had a project, android tv, in which one of the features was playing several videos at the same time, that is, the user looks at the screen and sees 4 videos. Then he chooses one of them and watches it already in full screen. The task is clear, it remains only to do. The peculiarity is that the video comes in the HLS format. I think that if you are reading this, you are already familiar with HLS, but still briefly - we are given a file, in which there are links to several streams that should change depending on the current Internet speed.
Example:
#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=688301 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0640_vod.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=165135 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0150_vod.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=262346 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0240_vod.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=481677 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/0440_vod.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1308077 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/1240_vod.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1927853 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/1840_vod.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2650941 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/2540_vod.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=3477293 http://qthttp.apple.com.edgesuite.net/1010qwoeiuryfg/3340_vod.m3u8 The first thing we started to implement this feature is the EXOPlayer cherer. Which was quite logical, since EXOPlayer uses hardware codecs to play the video stream. But it turned out that EXO has its dark side. When using EXO to run more than one thread, no one knows what will happen. In our case, when we launched 4 threads, on some devices everything worked well, on some only 3 worked, and the fourth one did not start, and on some, for example, something else happened on Nexus 7 2013. When Nexus 72013 ran more than 1 stream, the hardware codecs just crashed and no videos worked, not only in our application, but also in other applications that use hardware codecs. The only way to raise them is to reboot the device. As it turned out, this topic was devoted to the githaba topic . As it became clear, we cannot use hardware codecs, which means that we need to use software codecs and I remind you that the main task was to play 4 videos at the same time.
And the search began and we searched for a long time and we tried a lot, but the only thing that suited us was IJKPlayer . This is a player that is a ffmpeg wrapper. He played HLS, played them in 4 streams, and also played other streams that EXOplayer did not play on all devices (for example, HEVC). And for a very long time everything was fine, until we began to notice that the player always plays the same stream and does not change it depending on the network bandwidth. For the small videos, the preview was not a problem, but for full screen it was a problem.
After searching, it turned out that the streams do not change, and the host IJKPplayer advised to parse the streams separately from the player and launch the one that is needed (just like the ffmpeg ticket). Naturally, this did not fit because the player must adapt itself to the Internet. The problem is a problem, but it must be solved. There was nothing on the Internet to find so that it was decided to personally add logic to changing the flow to lib. But before you do something, you need to understand where to do it. FFMPEG itself is a very big way and it’s not so easy to understand what is what, but I outlined for you a few basic places that we will need to work with.
')
So, the main points that we need to know:
- There is a read_data method, which is in libavformat / hls.c, where the main magic happens. Here we download the stream and put it in the buffer. And at the end of the method there is a goto restart, where the segment changes. Before this restart, we will replace the stream if we need it.
- The second object that interests us is libavformat / avio.c. There is a ffurl_close method that is called when the link is closed, which means we’ll summarize the current throughput here. As well as the ffurl_open method, which, of course, opens our stream, which means we will reset the loaded data counter here, as well as restart the timer.
- It would also be nice to draw your attention to the new_variant and new_playlist methods - they create a playlist with all possible bitrates. According to my observations, the player takes the first item from the list and plays it, if there was any mistake, it takes the second item. If you need to make it so that only the smallest is played (which is logical, if you play 4 streams at the same time) or the largest stream, then pay attention to these methods.
So, summarize our tasks:
- Calculate current throughput
- Replace the link, if necessary, for the appropriate bandwidth
- Clean the data after the user stops watching the video
Listing:
bitrate_manager.h
#include <stdint.h> #ifndef IJKPLAYER_TEST_H #define IJKPLAYER_TEST_H extern int64_t start_loading; extern int64_t end_loading ; extern int64_t loaded_bytes; extern int64_t currentBitrate; extern int64_t diff; // extern char** urls; // , extern int64_t* bandwidth; extern int n_arrays_items; extern char* selected_url; extern int current_url_index; extern int64_t current_bandwidth; void saveStartLoadingData(); int64_t getStartLoading(); // int isInited(); // , void addToLoadingByte(int64_t bytesCount); // , void endOfLoading(); // void calculateAndSaveCurrentBitrate(); int64_t getDiff(); int64_t getLoadedBites(); int64_t getEndLoading(); int64_t getCurrentBitrate(); void setFullUrl(char* url); void setParturlParts(); // int doWeHaveBadwidth(); // void createDataArrays(int n_items); // void addData(int i, char* url, int64_t band_width); // void freeData(); // char* getCurrentUrl(); // int compareUrl(char* url); // void findBestSolutionForCurrentBandwidth(); char* getUrlString(int index); #endif //IJKPLAYER_TEST_H bitrate_manager.c
#include "bitrate_manager.h" #include <time.h> #include <stdint.h> #include <string.h> #include "libavutil/log.h" static const int64_t ONE_SECOND= 1000000000LL; int64_t start_loading; int64_t end_loading ; int64_t loaded_bytes; int64_t currentBitrate; int64_t diff; char** urls; int64_t* bandwidth; int n_arrays_items; char* selected_url; int current_url_index; int64_t current_bandwidth; /* * It conyains current last index + 1 */ int pointerAfterLastItem; int isInitedData = 0; int64_t now_ms() { struct timespec now; clock_gettime(CLOCK_MONOTONIC, &now); return (int64_t) now.tv_sec*1000000000LL + now.tv_nsec; } void saveStartLoadingData(){ loaded_bytes = 0LL; start_loading = now_ms(); } int64_t getStartLoading(){ return start_loading; } int isInited(){ return isInitedData; } void addToLoadingByte(int64_t bytesCount){ loaded_bytes += bytesCount; } void endOfLoading(){ end_loading = now_ms(); diff = end_loading - start_loading; } void calculateAndSaveCurrentBitrate(){ if(loaded_bytes != 0) { currentBitrate = loaded_bytes * ONE_SECOND / diff; } loaded_bytes = 0; } int64_t getDiff(){ return diff; } int64_t getLoadedBites(){ return loaded_bytes; } int64_t getEndLoading(){ return end_loading; } int64_t getCurrentBitrate(){ return currentBitrate; } int doWeHaveBadwidth(){ if(bandwidth && pointerAfterLastItem != 0){ return 1; } return 0; } void createDataArrays(int n_items){ isInitedData = 1; pointerAfterLastItem = 0; n_arrays_items = n_items; bandwidth = (int64_t*) malloc(n_items * sizeof(int64_t)); urls = (char**) malloc(n_items * sizeof(char*)); for(int i =0; i < n_items; i++){ urls[i] = (char*) malloc(sizeof(char)); } } void addData(int i, char* url, int64_t band_width){ if(band_width == 0LL){ return; } free(urls[i]); urls[i] = (char*) malloc(strlen(url) * sizeof(char)); strcpy(urls[pointerAfterLastItem], url); bandwidth[pointerAfterLastItem] = band_width; pointerAfterLastItem++; } void freeData(){ if(isInitedData == 0){ return; } isInitedData = 0; for(int i = 0;i < pointerAfterLastItem;++i) free(urls[i]); free(urls); free(bandwidth); } char* getCurrentUrl(){ return selected_url; } int compareUrl(char* url){ if(selected_url){ return strcmp(selected_url, url); } return 0; } void findBestSolutionForCurrentBandwidth() { if (currentBitrate == 0) { selected_url = urls[0]; current_url_index = 0; current_bandwidth = bandwidth[0]; return; } if (currentBitrate == current_bandwidth) return; int index = 0; int64_t selectedBitrate = bandwidth[index]; int start = 0; int length = pointerAfterLastItem; for (int i = start; i < length; i++) { if (currentBitrate >= bandwidth[i] && selectedBitrate <= bandwidth[i]) { index = i; selectedBitrate = bandwidth[i]; } } if (current_bandwidth != selectedBitrate) { selected_url = urls[index]; current_url_index = index; current_bandwidth = selectedBitrate; } } Now go to the listing of the ffmpeg
Add to avio.c
avio.c
int ffurl_open(URLContext **puc, const char *filename, int flags, const AVIOInterruptCB *int_cb, AVDictionary **options) { if(isInited() == 1) { saveStartLoadingData(); } …. } …. int ffurl_close(URLContext *h) { if( isInited() == 1) { endOfLoading(); calculateAndSaveCurrentBitrate(); } return ffurl_closep(&h); } In hls.c, the read_data method will look like this.
hls.c
static int read_data(void *opaque, uint8_t *buf, int buf_size) { struct playlist *v = opaque; HLSContext *c = v->parent->priv_data; // if (isInited() == 0) { createDataArrays(c->n_variants); for (int i = 0; i < c->n_variants; i++) { addData(i, c->playlists[i]->url, c->variants[i]->bandwidth); } } // , if(doWeHaveBadwidth() == 1 && isInited() == 1 && compareUrl(v->url) != 0){ strcpy(v->url, getCurrentUrl()); } int ret, i; int just_opened = 0; restart: if (!v->needed) return AVERROR_EOF; if (!v->input) { int64_t reload_interval; /* Check that the playlist is still needed before opening a new * segment. */ if (v->ctx && v->ctx->nb_streams && v->parent->nb_streams >= v->stream_offset + v->ctx->nb_streams) { v->needed = 0; for (i = v->stream_offset; i < v->stream_offset + v->ctx->nb_streams; i++) { if (v->parent->streams[i]->discard < AVDISCARD_ALL) v->needed = 1; } } if (!v->needed) { av_log(v->parent, AV_LOG_INFO, "No longer receiving playlist %d\n", v->index); return AVERROR_EOF; } /* If this is a live stream and the reload interval has elapsed since * the last playlist reload, reload the playlists now. */ reload_interval = default_reload_interval(v); reload: if (!v->finished && av_gettime_relative() - v->last_load_time >= reload_interval) { if ((ret = parse_playlist(c, v->url, v, NULL)) < 0) { av_log(v->parent, AV_LOG_WARNING, "Failed to reload playlist %d\n", v->index); return ret; } // if(isInited() == 1 && doWeHaveBadwidth() == 1) { addToLoadingByte(ret); } /* If we need to reload the playlist again below (if * there's still no more segments), switch to a reload * interval of half the target duration. */ reload_interval = v->target_duration / 2; } if (v->cur_seq_no < v->start_seq_no || v->cur_seq_no > (v->start_seq_no + (v->n_segments * 5)) ) { av_log(NULL, AV_LOG_WARNING, "skipping %d segments ahead, expired from playlists\n", v->start_seq_no - v->cur_seq_no); v->cur_seq_no = v->start_seq_no; } if (v->cur_seq_no >= v->start_seq_no + v->n_segments) { if (v->finished) return AVERROR_EOF; while (av_gettime_relative() - v->last_load_time < reload_interval) { if (ff_check_interrupt(c->interrupt_callback)) return AVERROR_EXIT; av_usleep(100*1000); } /* Enough time has elapsed since the last reload */ goto reload; } ret = open_input(c, v); // if(isInited() == 1 && doWeHaveBadwidth() == 1) { addToLoadingByte(ret); } if (ret < 0) { if (ff_check_interrupt(c->interrupt_callback)) return AVERROR_EXIT; av_log(v->parent, AV_LOG_WARNING, "Failed to open segment of playlist %d\n", v->index); v->cur_seq_no += 1; goto reload; } just_opened = 1; } ret = read_from_url(v, buf, buf_size, READ_NORMAL); // if(isInited() == 1 && doWeHaveBadwidth() == 1) { addToLoadingByte(ret); } if (ret > 0) { if (just_opened && v->is_id3_timestamped != 0) { /* Intercept ID3 tags here, elementary audio streams are required * to convey timestamps using them in the beginning of each segment. */ intercept_id3(v, buf, buf_size, &ret); } return ret; } ffurl_close(v->input); v->input = NULL; v->cur_seq_no++; c->cur_seq_no = v->cur_seq_no; // . bandwidth if(isInited() == 1 && doWeHaveBadwidth() == 1) { findBestSolutionForCurrentBandwidth(); if (compareUrl(v->url) != 0) { strcpy(v->url, getCurrentUrl()); } } goto restart; } Remaining trifles we add new files to the makefile inside libavformat in HEADERS and OBJS we add the corresponding references
makefile
NAME = avformat HEADERS = avformat.h \ avio.h \ version.h \ avc.h \ url.h \ internal.h \ bitrate_mamnger.h \ OBJS = allformats.o \ avio.o \ aviobuf.o \ cutils.o \ dump.o \ format.o \ id3v1.o \ id3v2.o \ metadata.o \ mux.o \ options.o \ os_support.o \ riff.o \ sdp.o \ url.o \ utils.o \ avc.o \ bitrate_mamnger.o \ We also add the IjkMediaPlayer_freeBitateWorkData method to ijkplayer_jni.c, which we will call after the view is complete, to clear the data.
ijkplayer_jni.c
static void IjkMediaPlayer_freeBitateWorkData(JNIEnv *env, jclass clazz){ freeData(); } // g_methods ... { "_freeBitateWorkData", "()V", (void *)IjkMediaPlayer_freeBitateWorkData }, ... Everything, our implementation is ready, now it remains to rebuild and watch videos with changing stream.
Source: https://habr.com/ru/post/307232/
All Articles