Someday in the future we will be able to wear glasses - and walk around the city, which is rendered in real time in the style we like. Sunlight or light haze, evening sunset, whatever. No matter how bleak the architecture is, it will be beautiful with glasses. Such rendering styles with the transfer of styles opens an amazing program Deep Photo Style Transfer , which is published in the public domain, as well as the scientific work underlying it (arXiv: 1703.07511).

Transferring the style from one image to another is an old problem that the developers have tried to solve relatively successfully in the past. Having correctly selected a sample for transferring a style, you can beautifully transform your photo - as if it was made at a different time of day, under different lighting, in different weather or in a special way artistically processed. Until now, style transfer techniques using neural networks were relatively limited — either they could only work with specific scenes, or the likelihood of style transfer suffered. The authors of the new program Deep Photo Style Transfer tried to solve all these problems.

The Deep Photo Style Transfer neural network is built according to the Neural Style transfer technique described by Gatis , but has been significantly reworked and improved. The main improvement is photorealism. In Gatis, even if the original image and the sample are photographs, the final image still resembles a drawing where the clear lines and borders become blurred and the textures are superimposed on neighboring objects. It looks beautiful, but not very real.
')

Color space only

The result of Deep Photo Style Transfer is strictly photorealistic. "Effects of painting" eliminated due to the ban on spatial distortion. Here the transfer of styles is limited only to the color space. In other words, the shapes of objects remain exactly the same as in the original. It is for this reason that in the future the “glasses”, which were mentioned at the beginning of the article, will become possible. All objects in the surrounding reality will retain their shape, just look in a different style.

The authors have achieved the task, using a specific layer of the neural network in the spirit of the Kirchhoff matrix (Laplacian matrix) - representation of the graph as a matrix. As the test showed on photographs with different subjects, this approach successfully suppresses distortion, while at the same time exerting minimal impact on the accuracy of photographs.

The most clearly described principle appears in two specially selected examples.



As you can see, the boundaries of objects are clearly preserved. Conversion occurs only in color space. In the first case, the style of fire is transferred to the perfume bottle, and in the second case, the texture of the apples changes according to the pattern.

Semantic segmentation

The second achievement of the authors of the program is to solve the problem of transferring the style between inappropriate objects in the original photo and the style sample. This is done through semantic segmentation. Therefore, the styles of houses are transferred only between houses, and the style of the sky only affects the sky. In many cases, semantic segmentation works very effectively, if the sample style contains the same semantic objects as the original image.

The above two approaches provided extremely realistic photo conversion.

Original


Style sample


Final image


Here are some more examples.

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

---

Original


Style sample


Final image

The quality of the work of such an algorithm can only be determined from the results of a survey of users who evaluate photorealism and the accuracy of the transfer of style. Such a survey would be conducted. He showed that the level of photo-realism Deep Photo Style Transfer is much higher than the previously presented development of Neural Style and CNNMRF, but inferior to the Piti color transfer system . According to the reliability of the transfer style Deep Photo Style Transfer is much better than other methods.



The authors of the scientific work intend to continue research in order to overcome the existing limitations of image segmentation. They also consider promising the direction of real-time photo processing with a pre-trained neural network.