Adaptation of the video to the form of a screen with a frequency of 1000 frames / s

The video is projected onto a moving sheet of paper (left) and onto the deformed fabric of the jersey

Everyone has seen the usual digital projector, which projects the video on a flat white surface - on the screen. Desirable in the dark. The screen requirements are very strict: the beauty of the picture depends on its quality. But imagine that a projector can project a video not only on a flat surface, but on a fabric of any shape, and moving as well ! This incredible technology was developed by Ishikawa Watanabe, a laboratory from the University of Tokyo.



According to Japanese engineers, the technological improvement of the projectors stopped in place. But this is an important technology that expands, complements the real world. Theoretically, projected images and videos can be very useful in different areas of augmented reality: from the entertainment industry to digital information assistants that display contextual information in a convenient place, on a glass of glasses, a helmet, or just a curtain in a room.
')
Most modern image projection technologies are limited to static surfaces. This significantly narrows the scope of their application.

Engineers from the Ishikawa Watanabe laboratory decided to remove this restriction, making the projectors more adapted to a dynamic world in which reality is constantly changing shape and transforming. In the augmented reality of the future, virtual objects and the real world will merge into a single organic form, perfectly adapted for human vision. Ideally, we should not distinguish between the inscriptions of paint and digital projections. In the more distant future, three-dimensional dynamic “living” objects should not differ much from real objects of the real physical world.

What is needed to bring this idea to life? First of all, a high-speed projector with an extremely high frame rate per second and low latency is required. It was with such requirements that the DynaFlash projector was developed in mind, which projects 8-bit images with a frame rate of 1000 frames / s and a delay of only 3 milliseconds.


Digital projector DynaFlash

The Japanese are not looking for easy ways. They did not stop at creating a high-speed projector, but went further - and figured out how to make it distort the image in accordance with the deformations of the flexible fabric almost in real time. What for? For example, so that the projector can project videos without distortion on moving objects. For example, on the jersey of a man in motion.

It is easy to guess that high-speed projection is performed through special tracking, that is, surface tracking. This is also an extraordinary task in the field of computer vision. The task of tracking surfaces that are creased is complicated by the fact that the surface can partially cover itself, not to mention the partial closure by foreign objects.

Of course, surface tracking should occur at the same high speed as the projector works. Therefore, Deformable Dot Cluster Marker technology has the same technical characteristics: 1000 frames / s and a delay of 3 milliseconds.

How to track the shape of the surface with such accuracy and speed? In principle, there are also a few options. The Japanese did this through a marker grid, which is applied to the object. The design of the marker grid is shown in animated graphics.



Tracking at high speed occurs on a highly parallelized problem in a variety of independent threads. However, even on a CPU, a performance of 1000 frames per second is achieved.


Marker recognition pipeline

After recognizing the marker grid, the program quickly calculates the necessary frame deformation vectors in the video - and projects the result in real time using the DynaFlash projector.


Calculation of vectors in the program

In general, according to the description, the system works quite logically and simply. The question is in technical implementation - how did they manage to optimize this machine vision task for such high performance?

The answer to this question can be found in the article " Dynamic Projection Mapping for Non-Rigid Surface Using Deformable Dot Cluster Marker ", which the authors published in IEEE Transactions on Visualization and Computer Graphics (doi: 10.1109 / TVCG.2016.2592910, pdf ). It describes in detail the algorithms for computing vectors for the marker grid, updating the positions of tracked point clusters, detecting false positives, interpolating lost clusters, etc. The authors write that they used the OpenMP package to parallelize the computations. The program's recognition of the marker network and frame-by-frame tracking required less than 2 ms and 1 ms, respectively, depending on the shape of the distortion. The table compares the result with another DRDM system that colleagues presented at the IEEE Symposium in 2011.



The authors are confident that the organic nature of augmented reality on the objects of the real world is an important condition for the integrity of impressions of this kind of synthetic reality. There should be no high latency in projecting video and geometric distortion.

The guys from the laboratory Ishikawa Watanabe have long been known for their unusual and interesting inventions. For example, in 2013, they invented the high-speed projector Lumipen , which projects video onto moving objects.



This technology can be used in augmented reality games, where people interact with virtual objects in reality, be it a pokeball or a soccer ball.

Virtual reality projection onto real-world objects is widely used in amusement parks around the world. It can be used in computer interfaces, prototyping, training with a teacher (for example, in driving instruction) and many other areas.