PhotoScan: how to take photos of photos without glare
April 19, 2017 we released the update of PhotoScan , our mobile application for iOS and Android, which allows us to digitize paper photos with a smartphone only. One of the key features of PhotoScan is the ability to remove glare from photos (they are often printed on glossy, well-reflective paper), as well as from pages of plastic albums or photos under glass. To do this, we have developed a unique combination of computer vision and image processing technology that can accurately combine and combine several slightly different photo shots and separate the highlight from the image itself in the photo.
One image is very difficult to automatically determine which parts of the image - this is the original photo, and which - the glare. Moreover, the flare can light the areas of the photo, because of which it will be impossible to restore that part of it that is hidden under the flare. But if you take a few photos of the photo, moving the camera, the location of the flare changes - it moves through different parts of the photo. In most cases, every single pixel will not be under the flare on at least one of the images. And although none of the images will not be flawless, we can combine several images of the printed photo taken at different angles, and thus remove the flare. The difficulty is that the images must be very precisely combined so that they are correctly combined, and this processing must be performed on the smartphone quickly enough to get the effect of an almost instant result.
')
Our technology was inspired by our previous work, published on SIGGRAPH 2015 , which we called “photography without interference” [ obstruction-free photography ]. It uses similar principles to remove various interferences from view. But the original algorithm was based on the generating model, in which the motion and appearance of the main scene and noise were evaluated. The capabilities of this model are great, and it is able to remove a variety of interference - but it is too expensive from a computational point of view, so that it can be used on smartphones. Therefore, we have developed a simplified model that relates to glare as an anomaly and tries to recognize the image under it. And although the model is simplified, it is still a very difficult task - recognition must be accurate and reliable.
We start with a set of photos taken by the user who moved the camera. The first shot - the “reference frame” - determines the desired final view of the picture. Then the user is recommended to take four additional shots. In each frame, we determine the key points (we calculate the properties of the ORB at the angles determined using the Harris algorithm ) and use them to determine homographs that correlate each subsequent frame with the reference one.
The technology seems straightforward, but it has a catch - homographs are able to match only flat images. But paper photos are often not flat (as in the example above). We use an optical stream - a fundamental representation of motion in computer vision, establishing a pixel-by-pixel correspondence between two images, correcting deviations from the plane. We start with frames aligned using homography, and calculate the “stream fields” for straightening images and further adjustments. Notice how, in the example below, the corners of the left photo “move” a little after defining the frames using only one homography. On the right you can see how the photo evened out better after applying the optical stream.
The difference is not very striking, but it significantly affects the final result. Notice how minor inconsistencies manifest themselves in the form of duplicate image fragments, and how these flaws are corrected by additional clarification using streams.
And in this case, the difficulty was to make the initially slow-working optical flow algorithm work faster on the smartphone. Instead of the traditional counting of flow for each pixel (the number of vectors is equal to the number of pixels), we build the flow field using fewer control points, and record the movement of each pixel in the image as a function of the movement of control points. Namely, we divide each image into non-intersecting cells that form a coarse-grained grid, and present the pixel flow in a separate cell as a bilinear combination of the flow and the four corners of the cell containing it.
Then it remains to solve a simpler problem, since the number of flow vectors is now equal to the number of lattice points, which are usually much smaller than pixels. This process is similar to image processing using splines . With this algorithm, we were able to reduce the optical stream calculation time on the Pixel phone by about 40 times!
Finally, to create a final image without a glare, for each area of ​​the frame we take the pixel values ​​and calculate the “soft minimum” to find the darkest of the options. Specifically, we calculate the expected minimum brightness for the available frames, and assign smaller weights to the pixels closer to the curved edges of the images. We use this method instead of directly calculating the minimum due to the fact that the same pixels may differ in different frames in different frames. A pixel-pixel minimum can lead to visible joints due to abrupt changes in intensity at the borders of overlapping images.
The algorithm is able to work in various scanning conditions - matte and glossy photos, photos in albums and without them, magazine covers.
To calculate the final result, our team developed a method that automatically determines the borders of the image and reduces it to a rectangular shape. Because of perspective distortions, a scanned rectangular photo usually turns into a non-rectangular quadrilateral. The method analyzes the image signals, color, edges to find out where the exact borders of the original photo go, and then applies geometric transformations to straighten the image. The result is a high-quality digital version of a paper photo without glare.
In general, a lot of things happen under the hood, and all this works almost instantly on your phone! You can try PhotoScan by downloading the version of the application for Android or iOS .
One image is very difficult to automatically determine which parts of the image - this is the original photo, and which - the glare. Moreover, the flare can light the areas of the photo, because of which it will be impossible to restore that part of it that is hidden under the flare. But if you take a few photos of the photo, moving the camera, the location of the flare changes - it moves through different parts of the photo. In most cases, every single pixel will not be under the flare on at least one of the images. And although none of the images will not be flawless, we can combine several images of the printed photo taken at different angles, and thus remove the flare. The difficulty is that the images must be very precisely combined so that they are correctly combined, and this processing must be performed on the smartphone quickly enough to get the effect of an almost instant result.
Hidden text
')
Our technology was inspired by our previous work, published on SIGGRAPH 2015 , which we called “photography without interference” [ obstruction-free photography ]. It uses similar principles to remove various interferences from view. But the original algorithm was based on the generating model, in which the motion and appearance of the main scene and noise were evaluated. The capabilities of this model are great, and it is able to remove a variety of interference - but it is too expensive from a computational point of view, so that it can be used on smartphones. Therefore, we have developed a simplified model that relates to glare as an anomaly and tries to recognize the image under it. And although the model is simplified, it is still a very difficult task - recognition must be accurate and reliable.
How it works
We start with a set of photos taken by the user who moved the camera. The first shot - the “reference frame” - determines the desired final view of the picture. Then the user is recommended to take four additional shots. In each frame, we determine the key points (we calculate the properties of the ORB at the angles determined using the Harris algorithm ) and use them to determine homographs that correlate each subsequent frame with the reference one.
Hidden text
The technology seems straightforward, but it has a catch - homographs are able to match only flat images. But paper photos are often not flat (as in the example above). We use an optical stream - a fundamental representation of motion in computer vision, establishing a pixel-by-pixel correspondence between two images, correcting deviations from the plane. We start with frames aligned using homography, and calculate the “stream fields” for straightening images and further adjustments. Notice how, in the example below, the corners of the left photo “move” a little after defining the frames using only one homography. On the right you can see how the photo evened out better after applying the optical stream.
Hidden text
The difference is not very striking, but it significantly affects the final result. Notice how minor inconsistencies manifest themselves in the form of duplicate image fragments, and how these flaws are corrected by additional clarification using streams.
And in this case, the difficulty was to make the initially slow-working optical flow algorithm work faster on the smartphone. Instead of the traditional counting of flow for each pixel (the number of vectors is equal to the number of pixels), we build the flow field using fewer control points, and record the movement of each pixel in the image as a function of the movement of control points. Namely, we divide each image into non-intersecting cells that form a coarse-grained grid, and present the pixel flow in a separate cell as a bilinear combination of the flow and the four corners of the cell containing it.
Then it remains to solve a simpler problem, since the number of flow vectors is now equal to the number of lattice points, which are usually much smaller than pixels. This process is similar to image processing using splines . With this algorithm, we were able to reduce the optical stream calculation time on the Pixel phone by about 40 times!
Hidden text
Finally, to create a final image without a glare, for each area of ​​the frame we take the pixel values ​​and calculate the “soft minimum” to find the darkest of the options. Specifically, we calculate the expected minimum brightness for the available frames, and assign smaller weights to the pixels closer to the curved edges of the images. We use this method instead of directly calculating the minimum due to the fact that the same pixels may differ in different frames in different frames. A pixel-pixel minimum can lead to visible joints due to abrupt changes in intensity at the borders of overlapping images.
The algorithm is able to work in various scanning conditions - matte and glossy photos, photos in albums and without them, magazine covers.
Hidden text
To calculate the final result, our team developed a method that automatically determines the borders of the image and reduces it to a rectangular shape. Because of perspective distortions, a scanned rectangular photo usually turns into a non-rectangular quadrilateral. The method analyzes the image signals, color, edges to find out where the exact borders of the original photo go, and then applies geometric transformations to straighten the image. The result is a high-quality digital version of a paper photo without glare.
In general, a lot of things happen under the hood, and all this works almost instantly on your phone! You can try PhotoScan by downloading the version of the application for Android or iOS .
Source: https://habr.com/ru/post/403389/
All Articles