We make a website for virtual reality. We build the monitor into the monitor and reflect on the future

Despite the fact that the concept of "virtual reality" is not the first year flashes before my eyes, it still remains a mystery to most people, and the prices of accessories associated with this entertainment can go to infinity. But there is a budget option. The average person who is interested in new technologies can afford a smartphone with a gyroscope, built-in Google Cardboard or any analogue of this simple device and a simple joystick with a pair of buttons. In our time, this option of exploring this technology is the most common. But, like many other technological innovations, like quadcopters, an interesting toy quickly turns into a dusting one on the shelf. Practical application is very limited. Adults who bought virtual reality glasses, the first days of playing with toys, watching various videos with beautiful young ladies , rejoicing like children. It's great. New impressions are always helpful. But quickly comes the understanding that the graphics in toys is at the level of the late 90s, the video is boring, and there is no other practical use of glasses for these people. For individual developers, designers and experimenters, glasses become a tool for work, but the average consumer puts them on the shelf and throws them. It would be interesting to apply this technology on the Internet on ordinary sites. In this article we will try using Javascript to adapt the layout we are used to under the new display conditions. There is not much information on the topic yet, the article is an experiment, so all interested people, as well as those who have had similar practical experience, are invited to join the discussion in advance and share their thoughts and experiences in the comments.


')

Introduction

Over time, new I / O devices appear in our world. The monitor, mechanical keyboard and mouse with a ball were supplemented with touchpads; . We stopped saying “do not poke a finger at the monitor” and began to think about the adaptation of interfaces to new realities. The buttons are getting bigger so that the finger does not miss, the interactive elements support svaypas and other movements with different numbers of fingers. With the advent of virtual reality glasses (a phone with a pair of lenses will also be considered as such), we should think about how to adapt what we have on the Internet at the moment to this turn of events. Yes, it will take several more years (maybe several decades) until such glasses are in every home, but if we don’t start thinking now, we’ll just postpone even more.

We get something, but we also lose something

VR glasses give us the opportunity to have information in three dimensions (or, at least, this effect is created), unlike a regular monitor. We have the opportunity to control what is happening by turning the head, which is quite interesting. Some joysticks, if you can call them that, allow you to control what we see with the help of hand movement in space. But at the same time we lose the ability to use the keyboard - only if you really know how to type blindly, you can use it. Moreover, for those who type a lot, the absence of the usual tactile sensation from it and the absence of a knock on the keys can be a serious challenge. A similar effect, although not to such an extent, occurs when switching from an old keyboard from the beginning of the 2000s to a new one, with a small key travel and almost silent. Losing the usual mouse - it is extremely problematic to imagine using a two-dimensional controller in three-dimensional space.



It turns out that we have a completely new set of input-output devices with a new behavior for us. We are not just evolving one of them, but replacing the entire stack. In such a situation, questions are inevitable about how well all these devices are adapted for solving already existing problems.

Content consumption. Still a plane?

Since we started talking about the application of technology from the point of view of the consumer, not the developer, we will continue to look from the same side. The consumption of information over the past decades has not changed significantly. Content can be presented in the form of text, sound, graphic image or video. Mechanical buttons are replaced by virtual ones, page turning is replaced by clicking on the links, but the content itself does not change much. Smells and tactile sensations from films or games to a wide range of consumers are not yet available.

With audio, everything is more or less clear. Headphones or several speakers can give a certain sense of volume, although two headphones do not allow you to "move" the sound source forward or backward. We can listen to music as before, and most of the interfaces work silently, so the question of sound will worry us last. Players for video in 3d are already in different versions. It can be considered that the issue of video consumption "full screen" is more or less resolved.

But with the text all the more interesting. First of all, once in the virtual space, the thought arises of moving away from the idea of ​​a flat monitor and positioning the text on a curved surface. Ball? And the user is in the center. Why not. But it is better to keep the vertical axes on the bottom, or even pull out from such an interface. Previously, we have not thought about this? And now it is necessary. Some toys make warnings for epileptics, and now they will probably be for people with problems with the vestibular apparatus. A cylindrical surface seems like a good choice. Equal “distance” to text at all points should reduce eye strain (compared to a huge monitor), efficient use of space ...



But reality beats very painfully. Because of the distortions, it is very difficult to read such text, and this is aggravated by the poor quality of the displays on most smartphones - the letters on the edges of the scope become a soap-pixel something. And constantly turning your head on every line of text is not an option. It seems that in the near future we will be limited to planes in 3d. We will have to lean towards the prism - we can embed three vertical monitors into our space and at the same time more or less preserve the readability of the text. Moreover, it is desirable to place them in such a way that when you turn your head to the “monitor”, your gaze falls exactly into its center and the line of sight is perpendicular to the plane of the monitor. Thus, the distortion will be minimal. If you tried to sit at an angle to the monitor, then you will understand what I mean. Why three? One could make a couple more monitors behind his back, but if one assumes a long work, then one would need to have the health of an astronaut in order to rotate on a chair the whole day. Some people and without glasses from a couple of turns will be enough for a rather unpleasant attack of dizziness and blurred eyes. So it makes sense to limit the viewing angle. Perhaps it is worth making experiments with a part of the dodecahedron, and not with a prism, but it is still difficult for me to imagine an interface built on pentagons.

And what about the input?

Three-dimensional controller - an interesting thing. But ... Have you ever seen an ordinary artist who draws on an easel? Formally, he is not engaged in physical labor - the brush weighs a few grams (well, maybe a couple of dozen), but after a full day of drawing his hand just falls off. And the back is tired, whatever they say. This is the same problem with 3D controllers. Hands weakened by many years of lying on the table and typing on the keyboard can not be suspended all day. So for games it’s great, for individual stages of 3d modeling, maybe, but for permanent work this kind of controller is not well adapted. There is an option when the hand lies and presses a few buttons. But this is already good. On button controllers, usually more than on a standard mouse (for users of top models from Razer, I remind you that there are two of them + a wheel).

With the keyboard - trouble. You can make a kind of on-screen keyboard, but the printing speed on such a miracle will be below the baseboard. Perhaps, over time, we will have semitransparent glasses spread so that we have an image at eye level, and we can see the real keyboard from below, but so far we are not talking about a budget version. Plus - for those devices that are on the market, there is a problem with fatigable eyes - you can only read a translucent text for a long time if there is something more or less monotonous and stationary in the background. You can see the notification, but read on an ongoing basis - no. Voice input is also far from perfect. It seems that in the near future to introduce large amounts of text in virtual reality will not work.

At the moment, the version with a smartphone will not allow us to combine glasses and things like WebGazer in order to use the direction of the look as a mouse substitute. If it is interesting, I will sort out separately the idea of ​​building an interface for a web application that can be controlled with just one glance (it is quite likely that this may have practical applications in medicine). But in the case of glasses, we only have a turn of the head. The most effective way to use it in our situation seems to be the following: take the center of the field of view, build a beam from the user and use its intersection with the monitor planes in order to create a similarity of a mouse on these planes.

Let's try to do something.

Since we make several monitors, we will create some markup for them.
(A link to GitHub with full sources will be at the end of the article). This is the usual markup, I used buttons and an accordion from one of my projects in full source code.

<div class='vr-area -left'> .... </div> <div class='vr-area -center'> .... </div> <div class='vr-area -right'> .... </div> 

There are not many options to use the usual HTML + CSS markup in three-dimensional space. But there are a couple of renderers for Three.js, which immediately attract attention. One of them, called CSS3DRenderer, allows us to take our markup and render it on a plane. This is quite suitable for our task. Second, the CSS3DStereoRenderer, which we will use, does the same, but divides the screen into two parts, creates two images, one for each eye.

Initialization of Three.js has already been mentioned in various articles more than once, so we will not dwell heavily on it.

 var scene = null, camera = null, renderer = null, controls = null; init(); render(); animate(); function init() { initScene(); initRenderer(); initCamera(); initControls(); initAreas(); initCursor(); initEvents(); } function initScene() { scene = new THREE.Scene(); } function initRenderer() { renderer = new THREE.CSS3DStereoRenderer(); renderer.setSize(window.innerWidth, window.innerHeight); renderer.domElement.style.position = 'absolute'; renderer.domElement.style.top = 0; renderer.domElement.style.background = '#343335'; document.body.appendChild(renderer.domElement); } function initCamera() { camera = new THREE.PerspectiveCamera(45, window.innerWidth/window.innerHeight, 1, 1000); camera.position.z = 1000; scene.add(camera); } 

The second component that we will immediately add is the DeviceOrientationController, which will allow the device to be rotated for orientation in 3d space. The technology of determining the position of the device is experimental, and, to put it mildly, today it works so-so. All three devices on which it was tested, behaved differently. This is further aggravated by the fact that in addition to the gyroscope (and, perhaps, instead of it), the browser uses a compass, which behaves very inadequately surrounded by various iron and wires. We will use this controller in order to preserve the possibility of using a conventional mouse when evaluating the results of an experiment.

 function initControls() { controls = new DeviceOrientationController(camera, renderer.domElement); controls.connect(); controls.addEventListener('userinteractionstart', function () { renderer.domElement.style.cursor = 'move'; }); controls.addEventListener('userinteractionend', function () { renderer.domElement.style.cursor = 'default'; }); } 

CSS3DRenderer, as we have said, makes it possible to use the usual markup in our 3d example. Creating CSS objects is a bit like making plain planes.

 function initAreas() { var width = window.innerWidth / 2; initArea('.vr-area.-left', [-width/2 -width/5.64,0,width/5.64], [0,Math.PI/4,0]); initArea('.vr-area.-center', [0,0,0], [0,0,0]); initArea('.vr-area.-right', [width/2 + width/5.64,0,width/5.64], [0,-Math.PI/4,0]); } function initArea(contentSelector, position, rotation) { var element = document.querySelector(contentSelector), area = new THREE.CSS3DObject(element); area.position.x = position[0]; area.position.y = position[1]; area.position.z = position[2]; area.rotation.x = rotation[0]; area.rotation.y = rotation[1]; area.rotation.z = rotation[2]; scene.add(area); } 

Then again comes the standard for almost all the Three.js demos functions for animating and changing the browser window.

 function initEvents() { window.addEventListener('resize', onWindowResize); } function onWindowResize() { camera.aspect = window.innerWidth / window.innerHeight; camera.updateProjectionMatrix(); renderer.setSize( window.innerWidth, window.innerHeight ); render(); } function animate() { controls.update(); render(); requestAnimationFrame(animate); } function render() { renderer.render(scene, camera); } 

But then the interesting begins. CSS3DStereoRenderer inherently creates two identical tree of elements, which with the help of CSS-transformations are built in the right place. In other words, there is no magic in this renderer. And what is there creates a problem. How can you interact with interactive elements if they are all duplicated? We can not focus on two elements at the same time, we can not do double: hover, in general a small ambush. The second problem is the id of elements that can only be in a single copy on the page. But if id can not be used, then you need to think hard about the interactions.



The only way to control the appearance of two elements at the same time is to add some CSS classes to them. We can also imitate click (). We can click on several elements, it does not lead to strange behavior of the browser. Thus, we can make the second cursor in the center of the field of view (not the entire screen, but in the centers of its halves, intended for different eyes). In order to recognize the element under the cursor, we can use the document.elementFromPoint (x, y) function. At this point, it seems very useful that the renderer uses transformations, but the elements themselves remain standard DOM elements. We do not need to come up with geometry with the intersection of the ray with the plane on which the content is rendered, searching for the coordinates of the intersection and determining the elements on the plane along these coordinates. Everything is much simpler. Standard function copes with this task.

The truth is, the question is about which element to add and delete classes to. I decided to temporarily stop at a simple transition to the parent elements until the focusable element appears. There is probably a more beautiful solution, and you can put it into a separate function, but I really wanted to see what happened there as soon as possible.

 function initCursor() { var x1 = window.innerWidth * 0.25, x2 = window.innerWidth * 0.75, y = window.innerHeight * 0.50, element1 = document.body, element2 = document.body, cursor = document.querySelector('.fake-cursor'); setInterval(function() { if (element1 && element1.classList) { element1.classList.remove('-focused'); } if (element2 && element2.classList) { element2.classList.remove('-focused'); } element1 = document.elementFromPoint(x1, y); element2 = document.elementFromPoint(x2, y); if (element1 && element2) { while (element1.tabIndex < 0 && element1.parentNode) { element1 = element1.parentNode; } while (element2.tabIndex < 0 && element2.parentNode) { element2 = element2.parentNode; } if (element1.tabIndex >= 0) { element1.classList.add('-focused'); } if (element2.tabIndex >= 0) { element2.classList.add('-focused'); } } }, 100); } 

We also add a quick imitation of clicking on both elements under our fake cursor. In the case of glasses, there should be a button processing on the controller, but not having one, let's stop at the usual enter on the keyboard.

 document.addEventListener('keydown', function(event) { if (event.keyCode === 13) { if (element1 && element2) { element1.click(); element2.click(); cursor.classList.add('-active'); setTimeout(function() { cursor.classList.remove('-active'); }, 100); } } }); 

What happened there?

Since a significant part of what was used in the article is experimental, it is still difficult to argue about cross-browser compatibility. In Chrome, under Linux, with a normal mouse, everything works, in Chrome, under an android, on one device everything turns slightly depending on the position in the room (I suppose because of the compass, which is influenced by ambient noise), on the other, everything flies into a bow. For this reason, I recorded a small video from the screen, which shows the result from the desktop browser:

Thoughts instead of conclusion.

The experiment is interesting. Yes, it is still a long way from real use. Browsers are still very poorly supported in determining the position of a device in space, the possibilities for calibrating the compass can be said to be absent (there is a formal compassneedscalibration event, but in practice there is not much use of it). But in general, the idea of ​​creating an interface in three-dimensional space is quite realizable. At least on the desktop, we can definitely use this. If you replace the stereo renderer with a regular one, and rotate the camera simply by mouse movement, you will get something similar to a regular three-dimensional toy, only with the ability to use bootstrap all the favorite tools of the layout designer.

Full example sources are available on GitHub.