Testing video codecs. Episode II: Encoders Attack
We continue to learn the secrets of testing video codecs. This time we will talk about encoders.
Link to the first part.
Just want to note that the encoders are not decoders, there are no predefined sets of video sequences to be tested on, which complicates the whole process from the very beginning. But there are commonly used, usually on them prove the superiority of some codecs (or their implementations) over others.
If we talk about comparison, we can distinguish two criteria: quality (subjective or objective) and performance. A rare codec is able to win in both categories at once. However, it will be about testing encoders.
So, at the entrance we have a decompressed video and the implementation of some codec (the actual encoder), the output is a compressed clip. We have already talked a little about the input data above, but I would like to note that we are limited only by our imagination and ... licenses. Those. You can always shoot the video yourself and then use it to mock the decoder / encoder, and you can not always download movies from the Internet for the same purposes. In addition, it can be useful to create artificial sequences: noises, frames of the same color, etc.
After a long process of compression, we will obtain a verified compressed stream with strict specifications. It has almost everything that was in the source data (we are talking about lossy compression). This is the file we are to investigate. And since we are talking about specifications, one of the important checks will be to make sure that our clip complies with the standard.
But the initiative is not always punishable: there is a desire - it is possible to disassemble the file into its own components and double-check everything. This should also be done in order to make sure that our encoder recorded all the parameters as we wanted, and not as horrible. Of course, in this case we are limited to “handles”, which we can “pull” in the encoder settings.
But back to our coded file. How to evaluate the quality of the picture that the user sees? Everything is very similar to what was in the first part: we need a decoder,
with the help of which we will get a decompressed video sequence, which we will be able to compare with the original with the help of the metrics already familiar to us - PSNR and SSIM.
However, here we come to the important question: what should be the thresholds of “similarity” of a video? In other words, what PSNR / SSIM values ​​will satisfy us and which ones won't? In general, good is when it is not bad, i.e. There are no artifacts, distortions and all the rest that is not so pleasing to the eye. And it's good if our metrics caught it. Do not forget to write down the desired tsiferku in a secret place and use it for the next check - and suddenly something broke. But here it can “break” in different ways.
So, let's say yesterday our encoder gave X dB at Y bitrate (we will encode with a constant bit rate)
Today, our encoder produces X1 dB at the same bitrate. What's the matter? Beat the alarm? Minute, and X1 more X? If so, then everything is fine, the encoder began to work better! Or not? Not everything is as simple as we would like: when using a constant bitrate, there is no guarantee that it really will be permanent - small deviations are possible (in both directions). Now imagine that the allegedly increased quality is due to the increased bitrate. And this means that there is no real improvement: the encoder “cheated” by taking more bits, which it used to improve the quality. Similarly, a decrease in the value of X1 compared to X does not indicate a deterioration of the encoder - maybe it saved us bits. Moral: no longer means better, less does not mean worse.
But what are we all about picture quality? It's time to think about the decoders, they then disassemble all this coded dregs. Did the encoder put all the necessary parameters into the stream? Did he put them right? Did all the requirements and regulations in the standard described?
And everything seems to be, you can finish on this, but there is one thing: we have the same SDK! This means that all the options for working with memory, threads, and everything else that the user can change, need to be checked. In theory, in most cases, all uses of the SDK should produce the same result.
So, we can very well test some of the most used model, and compare the rest with it. This will be faster, because decoding operations (for getting raw video) and counting metrics (just based on raw files) cannot be called cheap.
Summing up, I would like to note that while testing the decoder, we are most interested in the input data (and the more diverse, the better), then when testing encoders everything is important: both the input data and encoding parameters, which can be very, very much. And the checks themselves in the second case should be more intelligent.
')
UPD
The article about the review of approaches to testing Video Pre-Processing can be read on ISN
Link to the first part.
Just want to note that the encoders are not decoders, there are no predefined sets of video sequences to be tested on, which complicates the whole process from the very beginning. But there are commonly used, usually on them prove the superiority of some codecs (or their implementations) over others.
Remarkable comparisons are made by the guys from MSU, you can see it here: compression.ru
If we talk about comparison, we can distinguish two criteria: quality (subjective or objective) and performance. A rare codec is able to win in both categories at once. However, it will be about testing encoders.
So, at the entrance we have a decompressed video and the implementation of some codec (the actual encoder), the output is a compressed clip. We have already talked a little about the input data above, but I would like to note that we are limited only by our imagination and ... licenses. Those. You can always shoot the video yourself and then use it to mock the decoder / encoder, and you can not always download movies from the Internet for the same purposes. In addition, it can be useful to create artificial sequences: noises, frames of the same color, etc.
After a long process of compression, we will obtain a verified compressed stream with strict specifications. It has almost everything that was in the source data (we are talking about lossy compression). This is the file we are to investigate. And since we are talking about specifications, one of the important checks will be to make sure that our clip complies with the standard.
It is better to do it with third-party tools - so confidence in the assessment is higher for third-party users.
But the initiative is not always punishable: there is a desire - it is possible to disassemble the file into its own components and double-check everything. This should also be done in order to make sure that our encoder recorded all the parameters as we wanted, and not as horrible. Of course, in this case we are limited to “handles”, which we can “pull” in the encoder settings.
X264 lovers will have to put up with fewer switches
But back to our coded file. How to evaluate the quality of the picture that the user sees? Everything is very similar to what was in the first part: we need a decoder,
preferably third-party, reference to avoid double mistakes
with the help of which we will get a decompressed video sequence, which we will be able to compare with the original with the help of the metrics already familiar to us - PSNR and SSIM.
However, here we come to the important question: what should be the thresholds of “similarity” of a video? In other words, what PSNR / SSIM values ​​will satisfy us and which ones won't? In general, good is when it is not bad, i.e. There are no artifacts, distortions and all the rest that is not so pleasing to the eye. And it's good if our metrics caught it. Do not forget to write down the desired tsiferku in a secret place and use it for the next check - and suddenly something broke. But here it can “break” in different ways.
So, let's say yesterday our encoder gave X dB at Y bitrate (we will encode with a constant bit rate)
we will not take into account all other parameters, we will leave them fixed
Today, our encoder produces X1 dB at the same bitrate. What's the matter? Beat the alarm? Minute, and X1 more X? If so, then everything is fine, the encoder began to work better! Or not? Not everything is as simple as we would like: when using a constant bitrate, there is no guarantee that it really will be permanent - small deviations are possible (in both directions). Now imagine that the allegedly increased quality is due to the increased bitrate. And this means that there is no real improvement: the encoder “cheated” by taking more bits, which it used to improve the quality. Similarly, a decrease in the value of X1 compared to X does not indicate a deterioration of the encoder - maybe it saved us bits. Moral: no longer means better, less does not mean worse.
But what are we all about picture quality? It's time to think about the decoders, they then disassemble all this coded dregs. Did the encoder put all the necessary parameters into the stream? Did he put them right? Did all the requirements and regulations in the standard described?
All this and much more needs to be checked, double-checked, and then clarified. And so every time, with each encoder. But we have already talked about this a bit, we will not focus
And everything seems to be, you can finish on this, but there is one thing: we have the same SDK! This means that all the options for working with memory, threads, and everything else that the user can change, need to be checked. In theory, in most cases, all uses of the SDK should produce the same result.
Well, really, why should the quality suffer if the user uses D3D memory instead of the system one? Or the depth of asynchrony changed. Or it was decided to use more (or less) threads.
So, we can very well test some of the most used model, and compare the rest with it. This will be faster, because decoding operations (for getting raw video) and counting metrics (just based on raw files) cannot be called cheap.
Summing up, I would like to note that while testing the decoder, we are most interested in the input data (and the more diverse, the better), then when testing encoders everything is important: both the input data and encoding parameters, which can be very, very much. And the checks themselves in the second case should be more intelligent.
')
UPD
The article about the review of approaches to testing Video Pre-Processing can be read on ISN
Source: https://habr.com/ru/post/143393/
All Articles