Augmented Reality
Augmented reality (Augmented reality, AR) is a field of research focused on the use of computers to combine real-world and computer-generated data. An example from the cinema - shots taken from the point of view of the robot in the movie "Terminator". Directly on the image obtained from the eye-cameras, the data about the observed objects are displayed: “the policeman, the weight of such and such, the growth of such and such.”
To date, most of the research in the field of AR is focused on the use of live video, digitally processed and “enhanced” by computer graphics. The display of relevant additional information over the video can be observed, in particular, during the broadcast of sports competitions. Today, Formula 1 fans see not only race cars moving around the ring, but also information about the rider, team affiliation, position relative to cars of the most important rivals, and sometimes even graphs reflecting the number of engine revolutions.
More serious studies include the use of tracking the movement of objects, the recognition of coordinate marks using computer vision and the construction of a controlled environment consisting of an arbitrary number of sensors and power drives.
Initially, the term AR was introduced as opposed to virtual reality: instead of immersing a user into a synthesized, fully informational environment, the task of AR is to supplement the real world with additional information processing capabilities. Other researchers understand virtual reality as a special case of augmented reality. Augmented reality itself is a special case of a more general concept of mediated reality (med-R), in the sense that mediated reality allows you to consciously supplement or reduce, and also otherwise modify reality.
')
Ivan Sutherland, who built the working prototype of the system in 1967, can be considered the first researcher of augmented reality. He used Sword of Damocles stereo glasses to display three-dimensional graphics. The image in them was projected on two translucent glass mini-displays with silver coating. A curious name comes from the method of mounting the device - on the ceiling, which contrasted with the name of the class of this kind of equipment: Head-Mounted Display. For the first time, the system was used in a project executed in 1968 for the Bell Helicopter Company, in which the glasses were paired with an infrared camera located under the bottom of the helicopter. The camera was controlled by the movement of the pilot's head. Thus was born the concept of "augmented reality".
The current stage of research began in 1990, when researchers at Boeing decided to use head-mounted stereo displays when assembling and servicing aircraft, superimposing interactive graphics on real-world images.
One of the most famous researchers in this field today is Ronald Azuma of HRL Laboratories. In 1997, he published a large review article “A Survey of Augmented Reality”, where for the first time the problems and opportunities associated with the implementation of this technological concept were clearly outlined. Since 1999, the regularly held IEEE, ACM and Eurographics International Symposium on Mixed and Augmented Reality (ISMAR) conference has led its history. The most successful and well-known organizations specializing in augmented reality are located in Japan — the Mixed Reality Systems Lab — and Germany — the Arvika consortium.
Ronald T. Azuma (Ronald T. Azuma) defines AR as a system that:
Hermann H. Benes writes: “Augmented reality is defined according to the context and is not viewed as an abstract notation, but as if the objects of augmented reality exist in nature and life. Augmented reality is a tool that allows one or many observers to expand their field of vision with the help of virtual elements, usually created by a computer. The following rules can be defined as necessary for augmented reality to be accepted by business, education, and society:
Most people who are interested in AR, consider one of the most important characteristics of the way in which the transformation of the place where the interaction takes place. In an interactive system, it is important not only to accurately determine the location, but also to recognize the environment. Interaction is not just reading information from the screen, it is dissolving yourself in the surrounding space and objects. At the same time, the use of information systems is a conscious and cognitive act.
The three main components of a personalized augmented reality system are
Computer
Until recently, the main obstacle to practical implementation was the inability of mobile PCs to cheat three-dimensional graphics. Currently, the focus has shifted to developing adequate positioning and display methods.
Positioning
Using the now-classic GPS navigation system does not solve the problem. After all, the idea of ​​complementing reality is to get the right information in the right place. The “classic” GPS sensor gives a minimum error of 3 to 30 m. Considering that the signatures on the display must be combined with the image of real objects, such an error makes it pointless. And in the city GPS is often powerless.
Displaying video information is no less problematic. It is necessary to combine three-dimensional graphics and real objects, taking into account the movement of a person in space and his posture. Until now, mobile displays are the weak point of any wearable computers. Cheap, high-quality, lightweight and compact video glasses are still rare.
Since the main task of augmented reality is the synthesis of real and virtual objects in space, there is a need to pre-digitize data about the surrounding space. Registration of the geometric spatial characteristics of small rooms today has become the norm for a wide range of specialists. Everything turns out to be much more complicated when it comes to open spaces: how are virtual and real objects mutually located, which of them is in the foreground? The work here is carried out in two directions: shooting a “depth map” (depth sensing) in real time and preliminary collection of information about the terrain.
How to teach computer orienteering? The combination of a gyroscope and a compass gives quite good results, and if you add to them the recognition of images of previously known landscape elements, the accuracy increases to the pixel level. The discrepancy is measured in pixels solely because of the peculiarities of the human brain and vision, which can reveal the slightest mistake when placing virtual objects in real space.
Benefit here even bring the development of the creators of special effects in the movie, which are engaged in the restoration of the trajectory of the camera by tracking the movement in the frame of marker objects.
However, while the problem is even a delay in the implementation of conventional orientation algorithms. This should add the latency of the rendering engine. Scientists are forced to seek a solution in various two-dimensional methods, such as distortion and displacement of pre-rendered virtual objects that simulate motion in a plane and even rotation.
A great positive and stimulating influence on the development of augmented reality technologies was provided by the HiBall position tracking system. It was developed by staff at the University of North Carolina as part of a project funded by DARPA. Despite the fact that the system is able to work only with previously prepared closed rooms, the accuracy and minimum latency achieved during the experiments were record-breaking. The HiBall with a frequency of 1500 Hz registers any linear displacement from 0.2 mm and rotation through angles from 0.03 °.
The sensor itself, which supplies the monitored object or person, consists of six photodiodes and six lenses. Each sensor receives images from all lenses, giving a total of 36 independent views. Photodiodes capture signals from matrixed ceiling LED panels.
The main advantage of the system is its ability to automatically calibrate. After all, otherwise the installation of the panels would have resulted in a tedious procedure requiring precision precision. To prove the effectiveness of the adaptive properties of HiBall, the researchers deliberately introduced errors in the arrangement of the ceiling panels. 10 minutes after the start of use, having obtained the initial coordinates of the panels, the system managed to determine their new configuration.
The SCAAT (Single Constrain At A Time) algorithm allows to implement all this. As its name suggests, the idea is to use measurements as they are received. If a traditional system tries to simultaneously obtain all coordinates (x, y, z, t), then SCAAT fixes only one of them in one cycle, with each iteration imposing increasingly precise constraints on the solutions of the equation defining the probable location of the sensor in space. This minimizes latency and, at a sufficiently high measurement frequency, provides good accuracy. "The secret of the company" - in the original application of the Kalman mathematical filter (Kalman).
The culmination of the efforts of scientists was the commercial operation of the system - 3rdTech sells HiBall-3000 Tracker. One of the variants of the user-sensor device is a three-dimensional digitizer, which can digitize volumetric objects.
Alas, the HiBall traveler will not do. He needs a GPS. However, the accuracy of measurements is influenced not only by the relief and nature of the terrain, but even by atmospheric phenomena. And here comes to the aid Differential GPS .
On the ground, set the "base" GPS station, which are located in carefully selected reference points with known coordinates. The station’s task is to calculate the expected signal delay from all available satellites using the known coordinates and determine the difference between the estimated and actually obtained values. Then the radio list (usually at a frequency of 300 kHz) is given a list of satellites with correction factors for each of them. The mobile sensor in the vicinity can only accept the list, isolate the satellites that it is currently observing from it, and correct the measured time delay values ​​of the signals.
The next stage in the development of GPS positioning is Code-Phase GPS . A conventional GPS sensor uses code synchronization to determine the time delays of a satellite signal. The satellite retransmits a constantly repeating sequence of codes. The receiver, having received the signal, begins to cyclically shift its sequence until it coincides with the received one. The number of steps required for code synchronization gives an idea of ​​the time delay. But it is “giving an idea”: the frequency of the characters in the sequence is limited, and the accuracy of the measurements is also.
Code-Phase GPS sensors do not stop at code synchronization and, after the sequences coincide at the digital level, they begin to achieve a match between the carrier frequencies of the internal clock generator and the satellite signal. The transmission frequency of GPS satellites is 1.57 GHz, and if this value is translated into the distance language, then the maximum measurement accuracy for phase synchronization is about 3 mm.
Display
User interaction with the augmented reality system requires non-standard solutions. Of course, no one discards the usual keyboard and mouse, however, given the mobile nature of the technology, they are not an ideal match for video glasses. In experimental devices, researchers are trying to use almost the entire arsenal of information input techniques: manipulators with six degrees of freedom, speech recognition and gestures. But, as a rule, to fully interact with augmented reality, it is necessary to combine several devices.
The greatest interest is caused by “virtual interfaces”, which are striking in their simplicity and originality. Everything is exactly the same as during a children's game, when the ski pole turns into a sword, and the steering wheel torn from a children's avtomobilchik is an aircraft steering wheel. For example, a person picks up a regular board, and a computer “draws” on it the controls: keys, switches and displays. This method is implemented in the PIP (Personal Interaction Panel).
In another experimental system, the user rearranges the virtual furniture in a virtual toy apartment using a real small scapula. And in some modern slot machines you can play virtual tennis with a special tennis racket.
Delights the elegance of the Magic Book concept - a real book whose pages serve as "portals" to various virtual worlds. When a user, flipping through an album, decides to “enter” a certain world, his avatar appears on the corresponding page of books of other users of the system.
Displays (HMD or HWD, Head-Worn Displays) for augmented reality are divided into two main types: optical-transparent (optical see-through) and video-transparent (video see-through). The first ones allow a person to see the world around them - the viewer observes the generated image and the space around him. Transparent glasses use an external video camera to generate images of real objects.
In addition to video glasses, ordinary flat mobile and projection displays have also been used. The latter, in particular, are well suited for use in vehicles (automobiles, airplanes) and stationary systems (cranes, control panels for production processes, etc.). Finally, the most unusual option is to project images directly on objects around the world covered with a retro-reflective reflective layer. In this case, the reflection takes place strictly along the line of incidence of light, so several people viewing the same object from different points of view do not notice the “information” of their neighbors. With the help of retroreflective coverage, you can make objects transparent - for this you need to display the space behind them on their surface.
Unfortunately, optically transparent displays do not always allow to exclude / obscure real objects, and it is extremely difficult to achieve an exact coincidence of the virtual and real world. And video-transparent systems, in turn, suffer from discrepancies between the location of cameras and human eyes (parallax) - the picture is very far from the position from which he was used to seeing the world. In addition, the eternal problem of three-dimensional displays should be resolved - a clear relationship between pupil focusing and interpupillary distance. Depending on the distance of objects, both of these parameters consistently change, but when volumetric objects are projected on an equidistant plane from the eyes, the connection between them is broken, which leads to severe discomfort. This feeling is familiar to any visitor to IMAX 3D cinemas.
The complexity of the technological nature - this is not all. In augmented reality, there is a real threat of “frame congestion” - the threat of too much of the output information. Hundreds of significant objects sometimes come into view at the same time. To make life easier, the user has to filter the information and isolate the necessary information from it. Also, you should not allow virtual elements to overlap with objects of the real world that are important to the user: for example, virtual pointers to hotels and restaurants projected on the windshield of a car should not obscure the oncoming truck or traffic light.
Originating in the 60s of the XX century, augmented reality technology is experiencing rapid development. At the same time, developers face a number of problems and solve them quite successfully. In augmented reality, there is no distinction between the real and virtual worlds, which can make the world more interesting and richer. Thus, the technologies of augmented reality are gradually entering our life, changing and making it more comfortable. In addition, it is already not something unusual and exotic, we are confronted with it at almost every step, just do not notice it.
Leading Campaign Developers
www.arvika.de/www/e/home/home.htm - Arvika
www.artesas.de/site.php?lng=en - ARTESAS - Advanced Augmented Reality Technologies for Industrial Service Application
www.globis.ethz.ch/research/index - Global Information Systems Group
www.hitlabnz.org - HitLabNZ
www.iconolab.com - IconoLab
www.ipf.uni-karlsruhe.de —
www.mixedrealitylab.org — Mixed Reality Lab
www.nus.edu.sg —
www.t-immersion.com/home.asp — Total Immersion
www1.cs.columbia.edu/graphics/top.html — .
PS . - , .
To date, most of the research in the field of AR is focused on the use of live video, digitally processed and “enhanced” by computer graphics. The display of relevant additional information over the video can be observed, in particular, during the broadcast of sports competitions. Today, Formula 1 fans see not only race cars moving around the ring, but also information about the rider, team affiliation, position relative to cars of the most important rivals, and sometimes even graphs reflecting the number of engine revolutions.
More serious studies include the use of tracking the movement of objects, the recognition of coordinate marks using computer vision and the construction of a controlled environment consisting of an arbitrary number of sensors and power drives.
Initially, the term AR was introduced as opposed to virtual reality: instead of immersing a user into a synthesized, fully informational environment, the task of AR is to supplement the real world with additional information processing capabilities. Other researchers understand virtual reality as a special case of augmented reality. Augmented reality itself is a special case of a more general concept of mediated reality (med-R), in the sense that mediated reality allows you to consciously supplement or reduce, and also otherwise modify reality.
')
Ivan Sutherland, who built the working prototype of the system in 1967, can be considered the first researcher of augmented reality. He used Sword of Damocles stereo glasses to display three-dimensional graphics. The image in them was projected on two translucent glass mini-displays with silver coating. A curious name comes from the method of mounting the device - on the ceiling, which contrasted with the name of the class of this kind of equipment: Head-Mounted Display. For the first time, the system was used in a project executed in 1968 for the Bell Helicopter Company, in which the glasses were paired with an infrared camera located under the bottom of the helicopter. The camera was controlled by the movement of the pilot's head. Thus was born the concept of "augmented reality".
The current stage of research began in 1990, when researchers at Boeing decided to use head-mounted stereo displays when assembling and servicing aircraft, superimposing interactive graphics on real-world images.
One of the most famous researchers in this field today is Ronald Azuma of HRL Laboratories. In 1997, he published a large review article “A Survey of Augmented Reality”, where for the first time the problems and opportunities associated with the implementation of this technological concept were clearly outlined. Since 1999, the regularly held IEEE, ACM and Eurographics International Symposium on Mixed and Augmented Reality (ISMAR) conference has led its history. The most successful and well-known organizations specializing in augmented reality are located in Japan — the Mixed Reality Systems Lab — and Germany — the Arvika consortium.
Ronald T. Azuma (Ronald T. Azuma) defines AR as a system that:
- Combines the virtual and real
- Interacts in real time
- Works in 3D
Hermann H. Benes writes: “Augmented reality is defined according to the context and is not viewed as an abstract notation, but as if the objects of augmented reality exist in nature and life. Augmented reality is a tool that allows one or many observers to expand their field of vision with the help of virtual elements, usually created by a computer. The following rules can be defined as necessary for augmented reality to be accepted by business, education, and society:
- Full real-time interactivity
- Accurate and super fast tracking
- Stereoscopy
- Ultra-portable and wireless
- The feeling of "full immersion"
Most people who are interested in AR, consider one of the most important characteristics of the way in which the transformation of the place where the interaction takes place. In an interactive system, it is important not only to accurately determine the location, but also to recognize the environment. Interaction is not just reading information from the screen, it is dissolving yourself in the surrounding space and objects. At the same time, the use of information systems is a conscious and cognitive act.
The three main components of a personalized augmented reality system are
- wearable computer
- positioning tools
- display facilities
Computer
Until recently, the main obstacle to practical implementation was the inability of mobile PCs to cheat three-dimensional graphics. Currently, the focus has shifted to developing adequate positioning and display methods.
Positioning
Using the now-classic GPS navigation system does not solve the problem. After all, the idea of ​​complementing reality is to get the right information in the right place. The “classic” GPS sensor gives a minimum error of 3 to 30 m. Considering that the signatures on the display must be combined with the image of real objects, such an error makes it pointless. And in the city GPS is often powerless.
Displaying video information is no less problematic. It is necessary to combine three-dimensional graphics and real objects, taking into account the movement of a person in space and his posture. Until now, mobile displays are the weak point of any wearable computers. Cheap, high-quality, lightweight and compact video glasses are still rare.
Since the main task of augmented reality is the synthesis of real and virtual objects in space, there is a need to pre-digitize data about the surrounding space. Registration of the geometric spatial characteristics of small rooms today has become the norm for a wide range of specialists. Everything turns out to be much more complicated when it comes to open spaces: how are virtual and real objects mutually located, which of them is in the foreground? The work here is carried out in two directions: shooting a “depth map” (depth sensing) in real time and preliminary collection of information about the terrain.
How to teach computer orienteering? The combination of a gyroscope and a compass gives quite good results, and if you add to them the recognition of images of previously known landscape elements, the accuracy increases to the pixel level. The discrepancy is measured in pixels solely because of the peculiarities of the human brain and vision, which can reveal the slightest mistake when placing virtual objects in real space.
Benefit here even bring the development of the creators of special effects in the movie, which are engaged in the restoration of the trajectory of the camera by tracking the movement in the frame of marker objects.
However, while the problem is even a delay in the implementation of conventional orientation algorithms. This should add the latency of the rendering engine. Scientists are forced to seek a solution in various two-dimensional methods, such as distortion and displacement of pre-rendered virtual objects that simulate motion in a plane and even rotation.
A great positive and stimulating influence on the development of augmented reality technologies was provided by the HiBall position tracking system. It was developed by staff at the University of North Carolina as part of a project funded by DARPA. Despite the fact that the system is able to work only with previously prepared closed rooms, the accuracy and minimum latency achieved during the experiments were record-breaking. The HiBall with a frequency of 1500 Hz registers any linear displacement from 0.2 mm and rotation through angles from 0.03 °.
The sensor itself, which supplies the monitored object or person, consists of six photodiodes and six lenses. Each sensor receives images from all lenses, giving a total of 36 independent views. Photodiodes capture signals from matrixed ceiling LED panels.
The main advantage of the system is its ability to automatically calibrate. After all, otherwise the installation of the panels would have resulted in a tedious procedure requiring precision precision. To prove the effectiveness of the adaptive properties of HiBall, the researchers deliberately introduced errors in the arrangement of the ceiling panels. 10 minutes after the start of use, having obtained the initial coordinates of the panels, the system managed to determine their new configuration.
The SCAAT (Single Constrain At A Time) algorithm allows to implement all this. As its name suggests, the idea is to use measurements as they are received. If a traditional system tries to simultaneously obtain all coordinates (x, y, z, t), then SCAAT fixes only one of them in one cycle, with each iteration imposing increasingly precise constraints on the solutions of the equation defining the probable location of the sensor in space. This minimizes latency and, at a sufficiently high measurement frequency, provides good accuracy. "The secret of the company" - in the original application of the Kalman mathematical filter (Kalman).
The culmination of the efforts of scientists was the commercial operation of the system - 3rdTech sells HiBall-3000 Tracker. One of the variants of the user-sensor device is a three-dimensional digitizer, which can digitize volumetric objects.
Alas, the HiBall traveler will not do. He needs a GPS. However, the accuracy of measurements is influenced not only by the relief and nature of the terrain, but even by atmospheric phenomena. And here comes to the aid Differential GPS .
On the ground, set the "base" GPS station, which are located in carefully selected reference points with known coordinates. The station’s task is to calculate the expected signal delay from all available satellites using the known coordinates and determine the difference between the estimated and actually obtained values. Then the radio list (usually at a frequency of 300 kHz) is given a list of satellites with correction factors for each of them. The mobile sensor in the vicinity can only accept the list, isolate the satellites that it is currently observing from it, and correct the measured time delay values ​​of the signals.
The next stage in the development of GPS positioning is Code-Phase GPS . A conventional GPS sensor uses code synchronization to determine the time delays of a satellite signal. The satellite retransmits a constantly repeating sequence of codes. The receiver, having received the signal, begins to cyclically shift its sequence until it coincides with the received one. The number of steps required for code synchronization gives an idea of ​​the time delay. But it is “giving an idea”: the frequency of the characters in the sequence is limited, and the accuracy of the measurements is also.
Code-Phase GPS sensors do not stop at code synchronization and, after the sequences coincide at the digital level, they begin to achieve a match between the carrier frequencies of the internal clock generator and the satellite signal. The transmission frequency of GPS satellites is 1.57 GHz, and if this value is translated into the distance language, then the maximum measurement accuracy for phase synchronization is about 3 mm.
Display
User interaction with the augmented reality system requires non-standard solutions. Of course, no one discards the usual keyboard and mouse, however, given the mobile nature of the technology, they are not an ideal match for video glasses. In experimental devices, researchers are trying to use almost the entire arsenal of information input techniques: manipulators with six degrees of freedom, speech recognition and gestures. But, as a rule, to fully interact with augmented reality, it is necessary to combine several devices.
The greatest interest is caused by “virtual interfaces”, which are striking in their simplicity and originality. Everything is exactly the same as during a children's game, when the ski pole turns into a sword, and the steering wheel torn from a children's avtomobilchik is an aircraft steering wheel. For example, a person picks up a regular board, and a computer “draws” on it the controls: keys, switches and displays. This method is implemented in the PIP (Personal Interaction Panel).
In another experimental system, the user rearranges the virtual furniture in a virtual toy apartment using a real small scapula. And in some modern slot machines you can play virtual tennis with a special tennis racket.
Delights the elegance of the Magic Book concept - a real book whose pages serve as "portals" to various virtual worlds. When a user, flipping through an album, decides to “enter” a certain world, his avatar appears on the corresponding page of books of other users of the system.
Displays (HMD or HWD, Head-Worn Displays) for augmented reality are divided into two main types: optical-transparent (optical see-through) and video-transparent (video see-through). The first ones allow a person to see the world around them - the viewer observes the generated image and the space around him. Transparent glasses use an external video camera to generate images of real objects.
In addition to video glasses, ordinary flat mobile and projection displays have also been used. The latter, in particular, are well suited for use in vehicles (automobiles, airplanes) and stationary systems (cranes, control panels for production processes, etc.). Finally, the most unusual option is to project images directly on objects around the world covered with a retro-reflective reflective layer. In this case, the reflection takes place strictly along the line of incidence of light, so several people viewing the same object from different points of view do not notice the “information” of their neighbors. With the help of retroreflective coverage, you can make objects transparent - for this you need to display the space behind them on their surface.
Unfortunately, optically transparent displays do not always allow to exclude / obscure real objects, and it is extremely difficult to achieve an exact coincidence of the virtual and real world. And video-transparent systems, in turn, suffer from discrepancies between the location of cameras and human eyes (parallax) - the picture is very far from the position from which he was used to seeing the world. In addition, the eternal problem of three-dimensional displays should be resolved - a clear relationship between pupil focusing and interpupillary distance. Depending on the distance of objects, both of these parameters consistently change, but when volumetric objects are projected on an equidistant plane from the eyes, the connection between them is broken, which leads to severe discomfort. This feeling is familiar to any visitor to IMAX 3D cinemas.
The complexity of the technological nature - this is not all. In augmented reality, there is a real threat of “frame congestion” - the threat of too much of the output information. Hundreds of significant objects sometimes come into view at the same time. To make life easier, the user has to filter the information and isolate the necessary information from it. Also, you should not allow virtual elements to overlap with objects of the real world that are important to the user: for example, virtual pointers to hotels and restaurants projected on the windshield of a car should not obscure the oncoming truck or traffic light.
Originating in the 60s of the XX century, augmented reality technology is experiencing rapid development. At the same time, developers face a number of problems and solve them quite successfully. In augmented reality, there is no distinction between the real and virtual worlds, which can make the world more interesting and richer. Thus, the technologies of augmented reality are gradually entering our life, changing and making it more comfortable. In addition, it is already not something unusual and exotic, we are confronted with it at almost every step, just do not notice it.
Leading Campaign Developers
www.arvika.de/www/e/home/home.htm - Arvika
www.artesas.de/site.php?lng=en - ARTESAS - Advanced Augmented Reality Technologies for Industrial Service Application
www.globis.ethz.ch/research/index - Global Information Systems Group
www.hitlabnz.org - HitLabNZ
www.iconolab.com - IconoLab
www.ipf.uni-karlsruhe.de —
www.mixedrealitylab.org — Mixed Reality Lab
www.nus.edu.sg —
www.t-immersion.com/home.asp — Total Immersion
www1.cs.columbia.edu/graphics/top.html — .
PS . - , .
Source: https://habr.com/ru/post/47876/
All Articles