“We are starting! I said: let's start! ”, Or how we implemented the work with faststart-video for Android
Most owners of smartphones, tablets and other gadgets daily consume a huge amount of digital information, including media: images, music and, of course, video. On the last dwell in more detail. It is very important not to make users wait for content, especially when millions of people use the application every day. There is a lot of video content in the iFunny application we are working on, and we thought that downloading the whole video is long, uninteresting and not scalable. And what if in the near future you need to download a video that is not in 30-60 seconds, but in 5-10 minutes? Make the user wait half a minute while the video is downloading? And if the connection is bad? So the interest in the application is not long to lose. Therefore, we decided to make a faststart-video. Details under the cut.
So, the task is set, it's time to decompose it:
First, let's see what the faststart video is all about.
')
Immediately, I’ll make a reservation that the discussion will focus on the MP4 format (GIF and WebM are also supported), since basically the video in iFunny has this format. I think I will not reveal America to you if I say that the video file consists of fragments and can be decoded in parts. But how to decode it in parts if the metadata (Moov-atoms) of the video are in different parts of the file? This is where the faststart flag comes into play. With the help of it, all Moov-atoms at conversion are transferred to the beginning of the file. To perform this kind of operation using FFmpeg, you need to add faststart to the movflags, for example:
And after some simple manipulations, we will have a video ready for testing.
So, we figured out with faststart, it’s time to check if our player has support for playing this type of video.
We use Ijkplayer from bilibili , since at the time of the development of the first versions of the application, it provided much more functionality than the same ExoPlayer .
In the Ijkplayer documentation, nothing is written about the faststart video. But we already have implemented file playback logic. Then, looking at the
The developers of Ijkplayer made an interface with which we can “feed” video player parts in the form of an array of bytes, after specifying its final size.
From this it follows that we can render the video in parts, and this is exactly what we need. Now let's move from words to deeds.
Let's divide our subtask “Implement faststart playback” into 2 parts:
We need to simultaneously write to the file when downloading and read from it during playback, and also change the pointer in the file to play the video again. This feature is provided by the RandomAccessFile class, which means we will use it. To download use the library OkHttp. Could a situation arise when we have a half-downloaded video? Of course it can! So, we need somewhere to store information about the size of the downloaded part, as well as to know the final video size. Therefore, we will make a separate model for this purpose and call it MediaCache.
In this article you will not find how to manage this cache, since for each it will most likely be an individual implementation feature. Just say that we use our LRU cache. Read more about LRU here .
Let's go back to the download. We outlined the main points. Let's see how it looks in practice:
Very similar to the usual file download, is not it? That's just in parallel, the player will immediately read from this file. At this point, we move on to the second part of our subtask: let's make the player reproduce what we have. As it turned out, this is trivial: you just need to give the player bytes at the position that it asks for. In fact, there were only two dozen lines:
Fine! Everything worked, it remains only to finish a couple of small things, namely, to study the callbacks from the player and implement the progress bar display during video buffering / reloading. To do this, take a look at the IMediaPlayer interface, in which, besides the main methods, there are many constants.
Of all these constants, we only need 3 to implement / hide the progress bar:
You need to understand that the player can manually set the size of video buffering (we set it in a feature from the server), so a situation may arise when video rendering requires less bytes than the minimum buffer. That is, we need to take into account that rendering can start earlier than the callback with the end of buffering code is called. And for these callbacks, the OnInfoListener interface is responsible, which we need to implement:
That's all, our implementation is ready. Let's see what happened:
Comparison with average connection speed (Throttling 2 Mbps ADSL, Faststart on the right).
In this scenario, playback began in 5 videos during normal download versus 9 videos of faststart.
Comparison with a weak connection (Throttling 256 Kbps ISDN / DSL, Faststart on the right).
In this case, faststart is very different from normal playback: 6 seconds before the start of playback against 27!
So, the task is completed! The result speaks for itself. Thanks to this approach, we provided not only an almost instant video display, but also the ability to interrupt the download and cache it in parts, which is important. This gave us savings on returning to the incompletely downloaded video, since reloading is always faster than downloading the file again. And we spent the freed resources on additional prefetch of neighboring elements, if they were not loaded yet, in order to provide the user with content delivery as quickly as possible and without annoying progress bars.
This implementation method is not positioned as a silver bullet, especially here we looked at the faststart only on Ijkplayer, although on ExoPlayer it will most likely be similar. I hope this article will encourage you to start improving your product (even if it is perfect), because we all want to use the fastest and most convenient applications. Comments and suggestions will be waiting in the comments. Perhaps someone has already encountered this and he has something to say from the perspective of experience. Be sure to write, it is interesting to know how you coped with such tasks!
So, the task is set, it's time to decompose it:
- to study the structure and features of faststart-video;
- find out if our player supports this type of video;
- implement faststart playback;
- handle situations where download stops during playback.
First, let's see what the faststart video is all about.
')
Immediately, I’ll make a reservation that the discussion will focus on the MP4 format (GIF and WebM are also supported), since basically the video in iFunny has this format. I think I will not reveal America to you if I say that the video file consists of fragments and can be decoded in parts. But how to decode it in parts if the metadata (Moov-atoms) of the video are in different parts of the file? This is where the faststart flag comes into play. With the help of it, all Moov-atoms at conversion are transferred to the beginning of the file. To perform this kind of operation using FFmpeg, you need to add faststart to the movflags, for example:
ffmpeg -i <input> -c:v libx264 -c:a aac -movflags +faststart output.mp4 And after some simple manipulations, we will have a video ready for testing.
So, we figured out with faststart, it’s time to check if our player has support for playing this type of video.
We use Ijkplayer from bilibili , since at the time of the development of the first versions of the application, it provided much more functionality than the same ExoPlayer .
In the Ijkplayer documentation, nothing is written about the faststart video. But we already have implemented file playback logic. Then, looking at the
setDataSource(Context context, Uri uri) method setDataSource(Context context, Uri uri) , to which we pass the URI to the video file as a parameter, the question arose: what other implementations does it have? Can I just pass the URL to the video, and the player will figure it out himself? Can. But this method has limitations that are categorically not suitable for us: the inability to rewind and loop the video. After studying the implementations of the setDataSource method, we found the answer with the previously unexplored parameter setDataSource (IMediaDataSource mediaDataSource). Everything became clear after we looked inside the interface: public interface IMediaDataSource { int readAt(long position, byte[] buffer, int offset, int size) throws IOException; long getSize() throws IOException; void close() throws IOException; } The developers of Ijkplayer made an interface with which we can “feed” video player parts in the form of an array of bytes, after specifying its final size.
From this it follows that we can render the video in parts, and this is exactly what we need. Now let's move from words to deeds.
Let's divide our subtask “Implement faststart playback” into 2 parts:
- implement download video in parts;
- realize reading the video in parts and the ability to read from any part of the file.
We need to simultaneously write to the file when downloading and read from it during playback, and also change the pointer in the file to play the video again. This feature is provided by the RandomAccessFile class, which means we will use it. To download use the library OkHttp. Could a situation arise when we have a half-downloaded video? Of course it can! So, we need somewhere to store information about the size of the downloaded part, as well as to know the final video size. Therefore, we will make a separate model for this purpose and call it MediaCache.
data class MediaCache(val cacheFile: File) { var finalSize: Long = 0 var downloadedSize: Long = 0 val isCachePartiallyDownloaded: Boolean get() = downloadedSize != 0L && finalSize > downloadedSize } In this article you will not find how to manage this cache, since for each it will most likely be an individual implementation feature. Just say that we use our LRU cache. Read more about LRU here .
Let's go back to the download. We outlined the main points. Let's see how it looks in practice:
fun download(url: String) { val requestBuilder = Request.Builder().url(url) if (mediaCache.isCachePartiallyDownloaded) { requestBuilder.addHeader("Range", "bytes=${mediaCache.downloadedSize}-") } val response = mediaHttpClient.newCall(requestBuilder.build()).execute() // , 200 206 Range Range . val responseCode = response.code() if (responseCode == 200 || responseCode == 206) { handleResponse(response.body()) } else { // . } } override fun handleResponse(responseBody: ResponseBody?) { responseBody?.let { body -> if (mediaCache.finalSize == 0L) { mediaCache.finalSize = body.contentLength() } val file = RandomAccessFile(mediaCache.cacheFile, "rw") file.use { it.seek(mediaCache.downloadedSize) writeResponseInFile(body, it) } } } private fun writeResponseInFile(responseBody: ResponseBody, file: RandomAccessFile) { responseBody.use { val inputStream = BufferedInputStream(it.byteStream()) inputStream.use { stream -> var currentDownloadedSize: Long = mediaCache.downloadedSize val finalSize = mediaCache.finalSize val data = ByteArray(BYTE_ARRAY_SIZE) var count: Int while (currentDownloadedSize != finalSize) { count = stream.read(data) if (count != -1) { file.write(data, 0, count) currentDownloadedSize += count.toLong() mediaCache.downloadedSize = currentDownloadedSize } } } } } Very similar to the usual file download, is not it? That's just in parallel, the player will immediately read from this file. At this point, we move on to the second part of our subtask: let's make the player reproduce what we have. As it turned out, this is trivial: you just need to give the player bytes at the position that it asks for. In fact, there were only two dozen lines:
class CustomMediaDataSource(val mediaCache: MediaCache): IMediaDataSource { private val cacheFile: RandomAccessFile = RandomAccessFile(mediaCache.cacheFile, "r") @Throws(IOException::class) override fun readAt(position: Long, buffer: ByteArray, offset: Int, size: Int): Int { if (cacheFile.filePointer != position) { cacheFile.seek(position) } var count = cacheFile.read(buffer, offset, size) /** * -1, , : , . * , . * — 0, . */ if (count == -1 && position != mediaCache.finalSize) count = 0 return count } override fun getSize(): Long { return mediaCache.finalSize } override fun close() { try { cacheFile.close() } catch (e: Throwable) { Assert.fail(e.message) } } } Fine! Everything worked, it remains only to finish a couple of small things, namely, to study the callbacks from the player and implement the progress bar display during video buffering / reloading. To do this, take a look at the IMediaPlayer interface, in which, besides the main methods, there are many constants.
public interface IMediaPlayer { /* * Do not change these values without updating their counterparts in native */ int MEDIA_INFO_UNKNOWN = 1; int MEDIA_INFO_STARTED_AS_NEXT = 2; int MEDIA_INFO_VIDEO_RENDERING_START = 3; int MEDIA_INFO_VIDEO_TRACK_LAGGING = 700; int MEDIA_INFO_BUFFERING_START = 701; int MEDIA_INFO_BUFFERING_END = 702; int MEDIA_INFO_NETWORK_BANDWIDTH = 703; int MEDIA_INFO_BAD_INTERLEAVING = 800; int MEDIA_INFO_NOT_SEEKABLE = 801; int MEDIA_INFO_METADATA_UPDATE = 802; int MEDIA_INFO_TIMED_TEXT_ERROR = 900; int MEDIA_INFO_UNSUPPORTED_SUBTITLE = 901; int MEDIA_INFO_SUBTITLE_TIMED_OUT = 902; int MEDIA_INFO_VIDEO_ROTATION_CHANGED = 10001; int MEDIA_INFO_AUDIO_RENDERING_START = 10002; int MEDIA_INFO_AUDIO_DECODED_START = 10003; int MEDIA_INFO_VIDEO_DECODED_START = 10004; int MEDIA_INFO_OPEN_INPUT = 10005; int MEDIA_INFO_FIND_STREAM_INFO = 10006; int MEDIA_INFO_COMPONENT_OPEN = 10007; int MEDIA_INFO_MEDIA_ACCURATE_SEEK_COMPLETE = 10100; int MEDIA_ERROR_UNKNOWN = 1; int MEDIA_ERROR_SERVER_DIED = 100; int MEDIA_ERROR_NOT_VALID_FOR_PROGRESSIVE_PLAYBACK = 200; int MEDIA_ERROR_IO = -1004; int MEDIA_ERROR_MALFORMED = -1007; int MEDIA_ERROR_UNSUPPORTED = -1010; int MEDIA_ERROR_TIMED_OUT = -110; void setOnInfoListener(OnInfoListener listener); interface OnInfoListener { boolean onInfo(IMediaPlayer mp, int what, int extra); } void setDataSource(IMediaDataSource mediaDataSource); /* * , . */ } Of all these constants, we only need 3 to implement / hide the progress bar:
int MEDIA_INFO_VIDEO_RENDERING_START = 3; int MEDIA_INFO_BUFFERING_START = 701; int MEDIA_INFO_BUFFERING_END = 702; You need to understand that the player can manually set the size of video buffering (we set it in a feature from the server), so a situation may arise when video rendering requires less bytes than the minimum buffer. That is, we need to take into account that rendering can start earlier than the callback with the end of buffering code is called. And for these callbacks, the OnInfoListener interface is responsible, which we need to implement:
class InfoListener: IMediaPlayer.OnInfoListener { override fun onInfo(mp: IMediaPlayer, what: Int, extra: Int): Boolean { when (what) { IMediaPlayer.MEDIA_INFO_BUFFERING_START -> onBufferingStart() // IMediaPlayer.MEDIA_INFO_BUFFERING_END -> onBufferingEnd() // IMediaPlayer.MEDIA_INFO_VIDEO_RENDERING_START -> onBufferingEnd() } return true } } That's all, our implementation is ready. Let's see what happened:
Comparison with average connection speed (Throttling 2 Mbps ADSL, Faststart on the right).
In this scenario, playback began in 5 videos during normal download versus 9 videos of faststart.
Comparison with a weak connection (Throttling 256 Kbps ISDN / DSL, Faststart on the right).
In this case, faststart is very different from normal playback: 6 seconds before the start of playback against 27!
So, the task is completed! The result speaks for itself. Thanks to this approach, we provided not only an almost instant video display, but also the ability to interrupt the download and cache it in parts, which is important. This gave us savings on returning to the incompletely downloaded video, since reloading is always faster than downloading the file again. And we spent the freed resources on additional prefetch of neighboring elements, if they were not loaded yet, in order to provide the user with content delivery as quickly as possible and without annoying progress bars.
This implementation method is not positioned as a silver bullet, especially here we looked at the faststart only on Ijkplayer, although on ExoPlayer it will most likely be similar. I hope this article will encourage you to start improving your product (even if it is perfect), because we all want to use the fastest and most convenient applications. Comments and suggestions will be waiting in the comments. Perhaps someone has already encountered this and he has something to say from the perspective of experience. Be sure to write, it is interesting to know how you coped with such tasks!
Source: https://habr.com/ru/post/358248/
All Articles