Geekly Articles each Day
📜 ⬆️ ⬇️

How does Lytro, or another review


Lytro is a film camera with no focusing produced by Lytro ( Ren Ng startup - a graduate of Stanford). The camera works by the method of light fields, about the principle of operation of which I will try to tell (under the cut traffic!).
Pictures, where there are photos from lytro, are clickable (but the lytro site lately, sometimes, lies - probably it will fall now).

Camera

Full specs you can find on the site Lytro , about the appearance described in a previous review . The camera itself is made in China, connects and charges via USB:


The display is touch, of normal resolution, here it shows that the camera is charging:

')
The principle of operation is simple: top zoom (you need to hold your finger on the elastic band), on top of the same shutter button. The zoom is similar to 24-105mm.

Macro
If you bring the camera close to the object, it will not focus (yes, and after the shooting also does not focus):


To overcome this, you can enable the "creative mode". It is activated by clicking on the white square:


In creative mode, you can focus the camera by clicking on the touchscreen. He poked at the object, the camera thought and focused (autofocus, apparently, contrasting):


Focusing can be pretty close:


The limitation of creative mode is that after shooting the focus range will be limited.
At one time, manufacturers promised to do built-in focus stacking for the macro. Now there is no such feature in the camera, but this is easy to accomplish in Photoshop, getting the image from lfp - as you can see, the GRIP here is much more than before processing:


Settings
There are no more settings in the camera. The camera sets the exposure on its own; it is impossible to influence this process (I noticed that the metering in Matrix in EverydayMode is center-weighted in CreativeMode - but also not always). Here are a couple of examples:


In the worst conditions, it would be ISO 3200, 1/15 (although it is strange: with an accelerometer, it would be possible to determine that the camera is on a tripod and increase the exposure):


Soft

When you connect the camera, the software copies the photos from it, allows you to rotate them and upload them on lytro.com.
Written in Qt, stores information in a SQLite database, the format of which is described here .
Most recently, software came out under win, which is good news:


Format

Lytro has one .lfp format, used for both raw and web-optimized files. Consider it in more detail.
So .lfp is a container, divided into sections. Sections are of several types:

The size of the RAW file is 16MB, web-optimized 1MB. The latter is distinguished by the fact that it stores a set of pictures with a map that corresponds to the coordinates of the click and the number of the picture. A full byte description of the format can be found here .

RAW


The RAW file itself is the data from the matrix. We see the Bayer filter matrix, in front of which are located microlysis:


Treatment

The most interesting is processing. The process is fully described in the works of Ren Ng, I depicted it schematically:


First comes the demosai of the RAW file using traditional methods. After that, the light field is brought to such a state that the “cells” (micro lysis) fall into an integer number of pixels. To do this, you need to rotate the image slightly and scale it (this is the most difficult step that requires painstaking adjustment: ).
Then the light field is integrated with certain parameters and the output is a set of pictures with different focal lengths. I note that the software itself chooses where to focus and how many images you need to get: as long as there is no possibility to adjust this.
The resulting file is packaged and uploaded to the Internet. When a user clicks on one of the sections of the file (“focusing”), the application looks at the map, what kind of image it is and shows it. This is a depth map, but it is very rough, for practical purposes with such a resolution, you will most likely find it too small.

Post processing

Well, when you can fix something in Photoshop. It is possible here, but with a reservation.
Because The lytro format is described, it is easy to write a script that will be able to export all parts of the lfp file. Workflow is:
  1. Parse lfp;
  2. Editing pictures. Yes, Ivana Govnova will have to clean up not once, but exactly as many as the pictures in the container;
  3. We collect lfp back. Here we must take into account that it is necessary to count all sha1-hashes, including updating them in the TableOfContents-section, and then recalculate the hash from itself;
  4. Launch the Lytro application, which, when you click on this file, realizes that it has changed and updates its preview.

Voila, Lytro + Instagram Filter:

About light fields and plenoptic photo

In a classical camera, light coming from different points of the lens into one point of the matrix is ​​integrated and stored as the brightness value of a pixel at that point. In the plenoptic chamber, there is an array of microlenses in front of the matrix:


Each microlens distributes the light rays that come into it in different parts of the matrix:


The f / 2 aperture is not random. It allows the most economical use of the matrix. If you take less, there will be unused pixels on the matrix, if there are more, the projected images will intersect, which is unacceptable:


The light field in space is described by a function in five-dimensional space:



As you can see, the light field describes the energy brightness (radiance) for each point in space and for each direction of the beam from that point.
This function can be set in several ways:
On surface:


On the sphere:


Between two planes:


Just the last way to describe the light field is used in lytro: two planes are the plane of the back wall of the lens and the plane of the matrix.
The light field can be obtained using an array of cameras:


The plenoptic camera can be represented as a special case of an array of cameras: the role of the camera is performed by a microlens.
To obtain a photograph, it is necessary to integrate data from cameras (or in the case of lytro, data from microlens sections):


Depending on the method of selecting points within the window, there are two conversions:

The observer offset is the choice of a specific point inside the microlens:



Digital refocusing - the choice of the method of summable points from the sections of each microlens. Since there are few pixels under the microlens, in the case of lytro the possibilities of refocusing are very limited:



Theorem
Having a plenoptic camera with an aperture of f / N, in which each microlength corresponds to PxP pixels, you can get an image equivalent in sharpness obtained with a classical camera with an aperture of f / (N * P). For Lytro, it's about f / 14.
Another interpretation of the same theorem:
The plenoptic camera described in the theorem makes it possible to obtain images with a f / N depth of field focused in any place within the depth of field of a classic camera with an aperture of f / (N * P).


Handling difficulties
When processing there are several obstacles that complicate the writing of the visualizer of the light field:
  1. Microlenses are uneven. It is necessary to calibrate the algorithm;
  2. It is necessary to exclude the influence of vignetting. But remembering that under the microlens of only about 50 pixels, it becomes clear that this operation can produce a large error;
  3. Defective pixels. Like any other camera, they are here too. They need to be determined statistically so that they do not spoil the depth map;
  4. Performance. Calculations must be performed in the frequency domain using 4D Fourier transform.


Bokeh

Blur in the camera like this (these are pictures from one shot, pulled out of lfp):


For comparison, I made the same frame on lytro (left) and on the SLR f / 2 (right):


Artifacts

Almost all the images crawl out unnatural artifacts of various kinds. For example, such:


Or such:


Heartbreaking sight.

Tools to work with lytro

Lytro.Net : can open lfp and export their parts, there is also a console version:


Lytro compatible viewer : opens files, allows focusing on click, there is a focus distance slider:


Reflections

I could not find any tools that could work with light fields. All that is - work with. Lfp-format: display, export to jpeg.
After reading the works of Ren Ng, the technology no longer seems so mystical and unreal as it seemed at first.
The same file format with the same magic is definitely evil. I do not understand why they did it.
If you want to work with raw, I can send it to you (and I can sell the camera a little later).

Pictures

All the "live" pictures from the article are available on lytro.com :

The second day lytro site lies in the evening. Just in case, I uploaded lfp to file hosting , they open it up .

Links


About lytro:
Reverse Engineering the Lytro .LFP File Format - a simple, clear description of how-to-hack
lfptools - utility for working with lytro files on C
Lytro.Net - port lfptools on C #
Lytro meltdown - nui gui for working with lytro files on c #, SQLite structure of lytro SQLite, file format
The (New) Stanford Light Field Archive - examples of light fields from multiple cameras can be played online
Lytro Teardown - Lytro is disassembled here, you can see all the electronics

Ren Ng's work on light fields:
Light Field Sensing - light fields in pictures; not understand this description is difficult
Fourier Slice Photography - a theoretical description of the technique of light fields
Light Field Photography with a Hand-held Plenoptic Camera - how to apply the technique of light fields in photography
LightShop: Interactive Light Field Manipulation and Rendering - a little about how to present the information of the light field visually

Source: https://habr.com/ru/post/148497/

More articles:


All Articles