Possible use of virtuality for creating programs in the future
I believe that programmers can work more productively in virtual reality (VR). Why? First, the programmer will be able to work, being surrounded by a three-dimensional development environment instead of two-dimensional monitors. Secondly, interaction with BP is more natural and free than working with a mouse and keyboard.
I will describe the benefits of BP, two already existing prototype devices that can help programmers, and what awaits BP in programming.
If you are not familiar with BP: this is a big step towards enabling people to interact with the digital world in the way that is common in the physical world. This interaction uses the capabilities of the human body much more fully, which in principle can lead to more productive work, lower training costs and an increase in user satisfaction.
The stereoscopic image is a technique that is used in Oculus Rift -type VR glasses, when some excellent pictures are shown for each eye.
')
Thus, a person feels the depth of the virtual world, just like in the real world. Depth involves the spatial thinking of the brain. And this, in turn, should increase the productivity of work.
For example, the Data Mountain tool was created to store web bookmarks on an inclined plane. The creators thought that by adding depth, they involve spatial thinking, which will lead to an increase in speed. Microsoft conducted a study showing that "time to save and retrieve and the number of errors were reduced due to spatial thinking ."
A whole set of devices can read body movements. This gives a greater chance of self-expression, with the result that new opportunities appear.
Points like the Oculus Rift read the turns and head position. Leap Motion Controller is a small device that reads the movements of hands and fingers. The Kinect camera recognizes the movements of the legs, arms, and torso. The Virtuix Omni Omni-Directional Treadmill allows you to translate your movement into the movement of a game avatar. The Cyberith Virtualizer device also allows you to do jumps and squats.
Recently, an experiment was conducted on computer control through the brain, where participants were asked to imagine movement in BP, while scanning their brain. The researchers found that 70% could walk on a virtual street, just thinking about walking.
By bringing body movements into communication with the computer, users gain more freedom in this communication. And, as I will describe further, this additional freedom leads to very practical benefits, such as increased productivity.
One of the reasons why BP is useful is that they are easier to change than the physical environment. This means that tasks in BP can be completed faster and cheaper.
For example, American soldiers are trained in a battle simulator . Soldiers can at one point be trained in the mountains, and the next - into the desert. BP allows you to do it quickly and cheaply.
BP combines the best properties of the physical and virtual worlds, and provides a new means for combining them into one.
Here are some examples of existing BP prototypes that allow you to program and test code that demonstrate the benefits of BP.
RiftSketch is an environment for programming in BP that allows describing a three-dimensional scene in Three.js
The tree is created by a recursive algorithm, which is programmed by the user in a “floating” in virtual space editor. Birds are tweets that arrived through the Twitter API.
“Live coding,” which Stephen Tanimoto describes in his article , is a development environment that allows editing the program during its execution. The traditional development cycle includes 4 phases: editing, compiling, linking, launching. In live coding, there is only one phase. The program runs continuously even during editing.
The project allows you to quickly observe the changes made to the code. It is also claimed that the developer is easier to see where he made a mistake.
RiftSketch offers the user a simple text editor, flying in front of him in an empty BP. When the programmer begins to enter the code, the virtual world is updated and demonstrates a three-dimensional scene, depending on the code. The scene can be animated through a callback function. The state of the scene can be changed by setting the behavior of objects. Viewing it through immersion in BP makes the user feel inside the scene.
To facilitate the work with the keyboard, a webcam attached to the device transmits a picture from the real world, which is superimposed on the virtual one.
In addition, the project offers hotkeys and other systems for entering information, allowing you to quickly change the text of the program. With the help of gesture processing, a high speed of editing and changing the program parameters is achieved. For example, with gestures, you can quickly change the value of a digital parameter, which can indicate anything - from coordinates and rotation angle, to RGB components or object speed.
Imagine that a spacecraft has just landed on the surface of a comet. After 10 years of flight, the probe is in such a position that sunlight does not reach it. You have 24 hours before its batteries are completely drained. You need to change the program of the probe so that it can change its position. The satellite accompanying him collected all the necessary information about the surface on which the probe had landed.
You create a BP based on the information received and study the environment with a tool like RiftSketch. You go through all the possibilities, changing the program and watching how the simulation works. You bypass the BP around your probe after each attempt to make sure that he firmly stood on the supports and took a suitable position for further work. Finally, after a satisfactory simulation, you submit your program for review.
Immersion presents methods as code fragments and shows groups of fragments in the form of piles on the floor. A bunch can be deployed in a three-dimensional ring. The possibility of semantic increase of fragments makes the code readable, even when parts of the code are quite small.
The reviewer reads the code to change the position of the probe. On the left is a ring of code fragments. The model of the comet and the flight path are in the upper left. Top right shows the details of the code changes.
The reviewer initially sees the current code fragment in the center of the screen, while other related fragments lie in piles on the floor. Focusing on BP, the reviewer assesses the degree of the ratio of heaps of fragments to the current one according to how close they are located and by their size.
Immersion divides the floor into sections based on the program’s packages, and highlights them with a color that indicates how much the package has changed. Inspectors walking the BP can build improved visual models due to this spatial representation of the code. It should also make it easier to recall the details of the project from memory, since the brain works better in spatial environment.
The examiner can scrape a bunch of fragments and lift it up, turning it into a ring for closer examination. Now you can read the nearest fragment or scroll the ring to another fragment. You can move the study fragment up or down. When you need to undo an action, you can make a movement as if you are clearing the table.
Working in the same room, the team can communicate in person. Distributed try, but for now video conferencing is too slow, and instant messaging is too limited. High-speed data exchange, sufficient for the transfer of three-dimensional images, will provide the effect of presence. Startup High Fidelity is developing an infrastructure that will allow to transmit audio and video with a delay of no more than 90 milliseconds. In addition, their program can scan in real time a person’s face and recognize his emotions, transferring them to his avatar.
Other motion and body position sensors, Leap Motion and PrioVR, can also be used to transmit body language.
One of the advantages of collaboration is the possibility of pair programming, when two developers are working on the program simultaneously on the same computer, and can catch each other's mistakes. BP will allow such developers to work at one virtual workplace. View simultaneously file, talk and even show each other at code snippets.
Designers working together can communicate at the drawing board. A shared BP can provide a virtual room where everyone can see what others are doing. It will be possible to drag files or parts of code or graphic fragments, draw diagrams and program and database structures.
With the help of huge possibilities for simulation, BP opens up new perspectives for testing projects. You can go through the project of a factory or factory, try interaction with the simulation of a robot or a new car.
Devices like Microsoft HoloLens are designed to work in the real world, on which some virtual objects are superimposed - in augmented reality. DR expands the physical world, while BP immerses a person in a completely virtual one. What is more useful and more productive?
Video game controllers are useful for navigating and performing simple actions, and the keyboard allows you to enter more complex data. However, these devices require the user to interact simultaneously with both the real and the virtual world. Gesture recognition eliminates the need to work with the real world, but this can be extremely tedious. What is the best way to enter user data in BP?
Wearing glasses BP means disconnecting from physical reality, incl. from colleagues. This limits the possibilities of physical communication, and indeed the interaction with the real world. If you adjust the webcam to the glasses, it will help to organize the broadcast of the real world into the virtual one, but will limit the scope of visibility.
Some tasks are hard to imagine in 3D. For example, Immersion tries to represent two-dimensional pieces of code in 3D, but the code does not have a third dimension, therefore its representation is incomplete. In this area, you can begin to invent some kind of three-dimensional programming, as, in particular, is proposed in the work of Andy Ko .
Oculus Rift Development Kit 2 has a resolution of 1080p. The text can be read, but more is needed for normal programming. To use the Rift, you need to correctly measure the distance between the pupils, but the current tools are either long to use or inconvenient. Glasses have to suffer even more, setting up the Rift to work in glasses.
Two-dimensional environments for development do not use all the capabilities of the human body. BP is a step towards a development environment that can effectively link a three-dimensional image and body movements. With such advantages, new opportunities arise to create tools that were previously unavailable. These features provide increased productivity, more room for creativity and more convenience for the user.
Most likely to benefit from using BP are those developers and designers who create something in 3D. Two-dimensional image on the monitor severely limits their work. And other areas of development will also benefit from the use of 3D - but for this we need as many people as possible to create tools for the work of BP.
Virtuality is already here. What can you do to help her?
I will describe the benefits of BP, two already existing prototype devices that can help programmers, and what awaits BP in programming.
Benefits of BP
If you are not familiar with BP: this is a big step towards enabling people to interact with the digital world in the way that is common in the physical world. This interaction uses the capabilities of the human body much more fully, which in principle can lead to more productive work, lower training costs and an increase in user satisfaction.
Depth
The stereoscopic image is a technique that is used in Oculus Rift -type VR glasses, when some excellent pictures are shown for each eye.
')
Thus, a person feels the depth of the virtual world, just like in the real world. Depth involves the spatial thinking of the brain. And this, in turn, should increase the productivity of work.
For example, the Data Mountain tool was created to store web bookmarks on an inclined plane. The creators thought that by adding depth, they involve spatial thinking, which will lead to an increase in speed. Microsoft conducted a study showing that "time to save and retrieve and the number of errors were reduced due to spatial thinking ."
Body movements
A whole set of devices can read body movements. This gives a greater chance of self-expression, with the result that new opportunities appear.
Points like the Oculus Rift read the turns and head position. Leap Motion Controller is a small device that reads the movements of hands and fingers. The Kinect camera recognizes the movements of the legs, arms, and torso. The Virtuix Omni Omni-Directional Treadmill allows you to translate your movement into the movement of a game avatar. The Cyberith Virtualizer device also allows you to do jumps and squats.
Recently, an experiment was conducted on computer control through the brain, where participants were asked to imagine movement in BP, while scanning their brain. The researchers found that 70% could walk on a virtual street, just thinking about walking.
By bringing body movements into communication with the computer, users gain more freedom in this communication. And, as I will describe further, this additional freedom leads to very practical benefits, such as increased productivity.
Plasticity of the virtual environment
One of the reasons why BP is useful is that they are easier to change than the physical environment. This means that tasks in BP can be completed faster and cheaper.
For example, American soldiers are trained in a battle simulator . Soldiers can at one point be trained in the mountains, and the next - into the desert. BP allows you to do it quickly and cheaply.
BP combines the best properties of the physical and virtual worlds, and provides a new means for combining them into one.
Applications
Here are some examples of existing BP prototypes that allow you to program and test code that demonstrate the benefits of BP.
RiftSketch - live coding
RiftSketch is an environment for programming in BP that allows describing a three-dimensional scene in Three.js
The tree is created by a recursive algorithm, which is programmed by the user in a “floating” in virtual space editor. Birds are tweets that arrived through the Twitter API.
“Live coding,” which Stephen Tanimoto describes in his article , is a development environment that allows editing the program during its execution. The traditional development cycle includes 4 phases: editing, compiling, linking, launching. In live coding, there is only one phase. The program runs continuously even during editing.
The project allows you to quickly observe the changes made to the code. It is also claimed that the developer is easier to see where he made a mistake.
RiftSketch offers the user a simple text editor, flying in front of him in an empty BP. When the programmer begins to enter the code, the virtual world is updated and demonstrates a three-dimensional scene, depending on the code. The scene can be animated through a callback function. The state of the scene can be changed by setting the behavior of objects. Viewing it through immersion in BP makes the user feel inside the scene.
To facilitate the work with the keyboard, a webcam attached to the device transmits a picture from the real world, which is superimposed on the virtual one.
In addition, the project offers hotkeys and other systems for entering information, allowing you to quickly change the text of the program. With the help of gesture processing, a high speed of editing and changing the program parameters is achieved. For example, with gestures, you can quickly change the value of a digital parameter, which can indicate anything - from coordinates and rotation angle, to RGB components or object speed.
Usage example
Imagine that a spacecraft has just landed on the surface of a comet. After 10 years of flight, the probe is in such a position that sunlight does not reach it. You have 24 hours before its batteries are completely drained. You need to change the program of the probe so that it can change its position. The satellite accompanying him collected all the necessary information about the surface on which the probe had landed.
You create a BP based on the information received and study the environment with a tool like RiftSketch. You go through all the possibilities, changing the program and watching how the simulation works. You bypass the BP around your probe after each attempt to make sure that he firmly stood on the supports and took a suitable position for further work. Finally, after a satisfactory simulation, you submit your program for review.
Immersion - code check
Immersion presents methods as code fragments and shows groups of fragments in the form of piles on the floor. A bunch can be deployed in a three-dimensional ring. The possibility of semantic increase of fragments makes the code readable, even when parts of the code are quite small.
The reviewer reads the code to change the position of the probe. On the left is a ring of code fragments. The model of the comet and the flight path are in the upper left. Top right shows the details of the code changes.
The reviewer initially sees the current code fragment in the center of the screen, while other related fragments lie in piles on the floor. Focusing on BP, the reviewer assesses the degree of the ratio of heaps of fragments to the current one according to how close they are located and by their size.
Immersion divides the floor into sections based on the program’s packages, and highlights them with a color that indicates how much the package has changed. Inspectors walking the BP can build improved visual models due to this spatial representation of the code. It should also make it easier to recall the details of the project from memory, since the brain works better in spatial environment.
Interaction through gestures
The examiner can scrape a bunch of fragments and lift it up, turning it into a ring for closer examination. Now you can read the nearest fragment or scroll the ring to another fragment. You can move the study fragment up or down. When you need to undo an action, you can make a movement as if you are clearing the table.
Future of BP and programming
Remote Interaction
Working in the same room, the team can communicate in person. Distributed try, but for now video conferencing is too slow, and instant messaging is too limited. High-speed data exchange, sufficient for the transfer of three-dimensional images, will provide the effect of presence. Startup High Fidelity is developing an infrastructure that will allow to transmit audio and video with a delay of no more than 90 milliseconds. In addition, their program can scan in real time a person’s face and recognize his emotions, transferring them to his avatar.
Other motion and body position sensors, Leap Motion and PrioVR, can also be used to transmit body language.
Pair programming
One of the advantages of collaboration is the possibility of pair programming, when two developers are working on the program simultaneously on the same computer, and can catch each other's mistakes. BP will allow such developers to work at one virtual workplace. View simultaneously file, talk and even show each other at code snippets.
Pair development
Designers working together can communicate at the drawing board. A shared BP can provide a virtual room where everyone can see what others are doing. It will be possible to drag files or parts of code or graphic fragments, draw diagrams and program and database structures.
Simulation
With the help of huge possibilities for simulation, BP opens up new perspectives for testing projects. You can go through the project of a factory or factory, try interaction with the simulation of a robot or a new car.
Questions and problems of BP
Degree of immersion
Devices like Microsoft HoloLens are designed to work in the real world, on which some virtual objects are superimposed - in augmented reality. DR expands the physical world, while BP immerses a person in a completely virtual one. What is more useful and more productive?
Input forms
Video game controllers are useful for navigating and performing simple actions, and the keyboard allows you to enter more complex data. However, these devices require the user to interact simultaneously with both the real and the virtual world. Gesture recognition eliminates the need to work with the real world, but this can be extremely tedious. What is the best way to enter user data in BP?
Separation from the physical world
Wearing glasses BP means disconnecting from physical reality, incl. from colleagues. This limits the possibilities of physical communication, and indeed the interaction with the real world. If you adjust the webcam to the glasses, it will help to organize the broadcast of the real world into the virtual one, but will limit the scope of visibility.
3D translation
Some tasks are hard to imagine in 3D. For example, Immersion tries to represent two-dimensional pieces of code in 3D, but the code does not have a third dimension, therefore its representation is incomplete. In this area, you can begin to invent some kind of three-dimensional programming, as, in particular, is proposed in the work of Andy Ko .
Technical limitations
Oculus Rift Development Kit 2 has a resolution of 1080p. The text can be read, but more is needed for normal programming. To use the Rift, you need to correctly measure the distance between the pupils, but the current tools are either long to use or inconvenient. Glasses have to suffer even more, setting up the Rift to work in glasses.
Virtual reality is already here.
Two-dimensional environments for development do not use all the capabilities of the human body. BP is a step towards a development environment that can effectively link a three-dimensional image and body movements. With such advantages, new opportunities arise to create tools that were previously unavailable. These features provide increased productivity, more room for creativity and more convenience for the user.
Most likely to benefit from using BP are those developers and designers who create something in 3D. Two-dimensional image on the monitor severely limits their work. And other areas of development will also benefit from the use of 3D - but for this we need as many people as possible to create tools for the work of BP.
Virtuality is already here. What can you do to help her?
Source: https://habr.com/ru/post/249853/
All Articles