Expanding the canvas of Van Gogh's “Starry Night” with the help of the Wolfram Language (Mathematica)

Translation of the post Piotr Wendykier " Extending Van Gogh's Starry Night with Inpainting "
Download the translation in the form of a Mathematica document that contains all the code used in the article here (archive, ~ 8 MB).

Can computers learn to draw like Van Gogh? Definitely yes, to some extent! For this, like copyist artists, the algorithm will first need to take some original work, and then it will be able to create something based on them. How well can he handle it? Please judge for yourself.


Second Prize at the ZEISS Photo Contest

Recently, the Faculty of Engineering at the University of Cambridge announced the winners of the annual photo contest entitled “The Art of Engineering: Images from the Frontiers of Technology”. Second place went to Yarin Gal (Yarin Gal), a graduate student in machine learning, for his extrapolation of Van Gogh's Starry Night picture, which is shown in the image above. You can view this and similar “enhanced” computer images on the Yarin website Extrapolated Art . The creation of these artificial paintings used an extrapolation sketching algorithm called PatchMatch, and in this post I will tell you how you can create a similar effect using the Wolfram Language ( Mathematica ) language.
')
The term “digital inpainting” was first introduced in the article “ Image Inpainting ” at the SIGGRAPH 2000 conference. The main purpose of “sketching” is to restore damaged parts of images. However, this algorithm is also widely used to delete or move selected objects in images.

The Wolfram Language has a built-in function Inpaint , which implements this algorithm. The region to be “drawn” (retouched) can be given in the form of an image, a set of graphic primitives (vector graphics) or a matrix.

In [1]: =



Out [1] =



The Inpaint function can work with 5 algorithms, while you can specify a specific algorithm using the Method option, this is: “Diffusion,” “TotalVariation,” “FastMarching,” “NavierStokes,” and “TextureSynthesis” (default). “TextureSynthesis”, unlike other algorithms, does not operate separately with each color channel and does not create any new pixel color values. In other words, each pixel involved in “sketching” is taken from parts of the original image that have mask zero values ​​(that is, in other words, are black mask pixels). The example below clearly shows that the TextureSynthesis method is great for removing large objects from an image.

In [2]: =



Out [2] =



The “ Texture Synthesis” method is based on the algorithm described in the thesis “ Image Texture Tools ” (P. Harrison). This algorithm is an improvement of the best-fit method, which was proposed in 1981 in the doctoral thesis “ Computational Models for Texture Analysis and Texture Synthesis ” (D. Garber). The parameters of the “TextureSynthesis” algorithm can be specified using two suboptions: “NeighborCount” (default, 30) and “MaxSamples” (default, 300). The first parameter determines the number of nearby pixels that will be used in texture comparison, the second one sets the maximum sample size for the best texture selection.

Let's go back to the extrapolation of the Van Gogh paintings. To begin with, we import its image and remove the frame.

In [3]: =



Out [4] =



Now we need to expand the image field by adding a white field to it, and also create a mask for “sketching”.

In [5]: =



Out [8] =



Now we can extrapolate the image using the “TextureSynthesis” method.

In [9]: =



Out [9] =



Not so bad. You can get a different result by changing the values ​​of the “NeighborCount” and “MaxSamples” parameters.

In [10]: =



Out [10] =



You can also experiment with other parameter values ​​and other artworks. *

In [11]: =



Out [16] =



With the help of this technique you, I think, will be able to create an original gift for the New Year. From your personal photos or pictures you can make really interesting things. Or, you can surprise your child by showing him an improvisation on the subject of his drawing , as was done above based on the drawing of a 14-year-old girl. It all depends on your imagination!

Resources for learning Wolfram Language (Mathematica) in Russian: http://habrahabr.ru/post/244451