Cheap interpolation method works better than neural networks.

Compare RAISR with other advanced image resolution methods. For more examples, see accompanying research paper.

Increasing the resolution of images, that is, creating a high-resolution photo based on a single low-resolution photo is a very well-studied scientific problem. It is important for many applications: zooming photos and text, the projection of video on the big screen, etc. Even in films, detectives sometimes manage to look at the number of the car in the frame from a surveillance camera, “bringing” the photograph to the limit. And not only the car number. Everything is limited by the imagination and conscience of the director and screenwriter. They can bring the photo even closer - and see the reflection of the criminal in the rearview mirror or even in the polished metal head of the bolt that holds the license plate. Viewers like that.

In practice, the possibilities of such programs are much more modest. For example, on October 29, 2016, a Neural Enhance program was posted on GitHub, which improves the resolution of photos using a neural network. The program immediately entered the list of the most popular repositories for the week.


Neural Enhance Sample
')

Another example of the work of the program Neural Enhance, which is published in the public domain on GitHub

Employees of Google Research are also working in this direction - in the company's official blog yesterday they talked about the resolution increasing method, which they called RAISR (Rapid and Accurate Image Super-Resolution).

Historically, simple interpolators were used for interpolating images, which find the intermediate values ​​of new pixels by a known set of pixel values ​​of the original image. Different methods were used to calculate the average values: interpolation using the nearest neighbor method, bilenovy interpolation, the cubic method, the bicubic method, etc. These are all fairly simple math formulas. They were widely used in various applications because of their simplicity and unpretentiousness. They do not adapt to the image content at all, which often leads to the appearance of unpleasant artifacts - too vague fragments, characteristic aliasing distortions.

In recent decades, much more advanced interpolation programs and methods have been developed, which explicitly take into account the characteristics of the original image. They are able to use and scale fragments of the original image, fill in the sparseness, apply Gaussian mixtures . New methods have significantly improved the quality of interpolation (digital restoration of originals) by increasing the complexity of calculations.

Employees of Google used the method of machine learning on external samples. This method has gained great popularity in recent years and has been described in many scientific papers. The basic principle is to "predict" the content of the image in high resolution on its reduced copy. For such training, the standard method of teaching samples is used.

During RAISR training, a base of simultaneously generated pairs of images in high and low quality was used. Used pairs of small fragments of the image for the standard 2 interpolation, that is, fragments of 3 × 3 and 6 × 6 pixels. The learning and operation algorithm of the RAISR is shown in the diagram.





The following illustration shows the four global filters that were allowed to be applied at the training stage. Accordingly, the programs were trained to apply them most effectively, depending on the content of this particular fragment of several pixels.





Each type of filter operates for its own type of pixels: from P1 to P4, in accordance with the types of pixels that are used by the bilinear interpolation algorithm.



In some ways, the RAISR machine learning method is similar to neural network training. But in fact, it is an adaptation of various standard interpolation filters for each individual small fragment of the original image. That is, it is the same old “linear interpolation”, but as it were on steroids - without its inherent artifacts and with adaptation to the content of the image.

Comparative testing has shown that such an algorithm in many cases works even better than modern advanced interpolation methods based on neural networks (SRCNN on illustrations).



Moreover, such a hash-based method is much less resource-intensive and more practical in practice than the training and use of a neural network. The difference in performance is so large (10-100 times) that this program can be easily run even on ordinary mobile devices, and it will work in real time. Nothing prevents this filter from being implemented in modern interpolation applications of images on smartphones, including the Android camera application, which performs interpolation during digital zooming. It is possible that Google is exactly what is going to do in the first place. At least, this is an example of the most massive ubiquitous interpolation on millions of devices.


Left: low resolution original. Center: the result of the standard bicubic interpolator. Right: RAISR output

Photos will be better immediately with all Android users.

By the way, another interesting and important advantage of RAISR is that in the process of learning this program can be trained to eliminate characteristic compression artifacts, including JPEG. For example, on a mobile device, photos can be stored in a compressed form with artifacts, and displayed on the screen without artifacts. Or, the algorithm can be applied on Google photo hosting to automatically improve user photos, with the elimination of JPEG artifacts that are present almost everywhere.


Left: Low resolution original with JPEG-specific aliasing artifacts. Right - RAISR issue

A research paper by Google Research staff will soon be published in IEEE Transactions on Computational Imaging. (Note: The lead author of the research was an intern at Google Research during the preparation of the article, but now works at the Israeli Technion Research Institute of Technology).