VR / AR in the browser. How to quickly integrate and make your first application using WebVR API

Virtual and augmented realities are actively developing and are beginning to lay claim to the capture of the world. Moreover, the use of these technologies is diverse and is not limited to games only. And with the advent of tools for working with VR / AR technologies in the browser, interest has increased even more. Already, you can experiment and create MVP (Minimum Viable Product) web projects using the JavaScript API, which is called WebVR. Let's figure out what it is and how to work with it. And is it possible to do without this WebVR API itself?

This article is for informational purposes only and is the first in a series of articles about the WebVR API and Web AR implementations that I plan. If the topic comes, then I will continue to develop it, showing already more specific examples using various techniques and frameworks. I want to share my personal experience and discuss in the article the basic principles of working with WebVR, how to make Web AR, what to do if you have iOS, and consider the devices for which you can develop all this today.

In Tutu.ru, I work as a system architect in the Frontend division. I am engaged in various RnD (Research and Development) tasks. The VR / AR theme does not intersect directly with my work, but we are already thinking in the company about using this technology in the context of the tourism business. The technology of the future always attracts me and I really like the Frontend stack. Rather, I like the idea of doing everything in the browser. Let it still imposes a number of restrictions and gives some difficulties. But they are all solvable, and after some time they are completely leveled.
')

Where the legs grow

The speculation itself began to be written in 2014, but the first version was introduced in early 2016 by Mozilla. Then the first full-fledged draft specification of WebVR API was presented. This specification describes the software interface for working with various VR devices such as the Oculus Rift and the like (this spec is actually written originally under the oculus). This specification was developed with the participation of Brandon Jones (Brandon Jones) from Google and Justin Rogers (Microsoft).

Support for the new WebVR specification has already been implemented in Firefox and Mobile Chrome (more precisely, the degree of implementation and the nuances will be discussed later). And this means that today you can freely experiment and try the technology of tomorrow's future. If your browser is old, strange or not updated - there are polyphiles for all this.

Why do we need WebVR API?

WebVR API is a software interface for working with devices. He knows nothing about 3D graphics. Work with graphics, scene rendering, installation of light sources and everything else lies on the harsh shoulders of programmers. WebVR API only allows you to abstract access to devices. This API provides tools for rendering images, for obtaining information about the device, its capabilities and technical characteristics, but the picture itself and the 3D world need to be drawn using already familiar web technologies such as: HTML, CSS, WebGL, Canvas, etc ...

The WebVR API provides us with several basic objects for working:

Navigator - allows you to get a list of devices, determine the active;
VRDisplay - reports whether the headset is wearing a head, information about frames, eyes;
VRPose - information about the position and orientation of the device, the speed of movement and direction;
VREyeParameters - information on how to render video in each individual eye;
VRFrameData - information about the scene frame for projection on a single eye.

A complete list of objects can be obtained from the following sources:

developer.mozilla.org/en-US/docs/Web/API/WebVR_API
msdn.microsoft.com/en-us/library/mt806281 (v = vs.85) .aspx

Hello VR world

A simple boilerplate code for working with a JavaScript device on JavaScript looks like this:

navigator.getVRDisplays().then(displays => { if (displays.length < 1) { return console.warn('No displays found!'); } displays.forEach(vrDisplay => { buildVRDisplay(vrDisplay) }) }); function buildVRDisplay(vrDisplay) { //       const frameData = new VRFRameData; //         vrDisplay.getFrameData(frameData); //     const rightEye = vrDisplay.getEyeParameters('left'); const onAnimationFrame = () => { //     vrDisplay.requestAnimationFrame(onAnimationFrame); }; vrDisplay.requestAnimationFrame(onAnimationFrame); }

This code should give an idea of how the operation using the WebVR API looks. More detailed examples will be considered further, but now let's talk about browser support and devices.

What devices to work with?

When they talk about VR, they recall Oculus Rift and the like. If we are talking about WebVR, it is logical to assume that we are counting on developing for browsers.

The most affordable and popular devices today are the so-called Cardboard devices (or also called VRBoxes). Mobile VR device list can be listed on the fingers:

Google Cardboard;
Google Daydream (new performance of Cardboard devices);
various Chinese VRBox'y for all models of smartphones;
Samsung GR VR (in my opinion, a good combination of VR helmet and Cardboard device capabilities, and even access to the Oculus Store).

Noname VRBox

Just a box with lenses. There are very different versions, up to copies of the Samsung GR VR, but with the connection via Bluetooth of any phone model. They are made in different form factors and in different versions. You can buy a Bluetooth joystick that will work with any smartphone on any OS. The quality is very good, there are no complaints about the lenses. All the salt lies in the smartphone, the size of the display and the screen resolution.

Xiaomi vrbox

Features: the presence of a mechanical button-manipulator, working on the principle of the stylus, which taps into the “blind zone” of the screen in the bridge of the nose (engineers burn :)). Zip fastener.

Samsung GR VR

Features: connects to the box via USB, the presence of the touch panel and buttons on the helmet. At the time of connection, the Oculus functionality is activated, and the phone pretends to be an Oculus-compatible device.

Google cardboard

Features: the presence of the button, made in the form of a small round magnet. The application reads the change in the magnetic field by the built-in magnetometer, and this change is counted as a button action in Cardboard-compatible devices.

Google daydream

Features: the main difference from the Cardboard is a more thorough and convenient headset and the presence of a gamepad. This is a full-fledged VRBox, not cardboard :)

Here we will talk about them. If you have a smartphone with a Chrome browser and some variation of VRBox, then you can already check in WebVR action. The best option is Chrome Canary or Samsung Internet. Of course, I must say about the exception - this is iOS. But there VR can be used through polyphiles, so demos will also be available from iOS devices (depends on the implementation). Again, you need to understand that the WebVR API is not about 3D graphics, and you can make the WebVR world under iOS without using the VR API itself (well, or with polyfiles).

You can also look at the VR world through the desktop browser window, which usually develops. First, the world is built without helmets, and then the possibility of splitting the image under an optical pair is added. To do this, use either Firefox Nightly or Chrome with this plugin: WebVR API Emulation. Either Chrome Canary with the included support, or a special Chromium assembly ... Well, you get the idea :)

What with support in browsers?

So, for today WebVR API is to some extent supported in the following browsers:

Microsoft Edge in Hololens;
Mozilla Servo on HTC Vive;
Firefox Nightly;
Chrome Canary (you need to enable the chrome flag : // flags / # enable-webvr );
Chrome Canary for Android;
Chrome for Android (including Daydream devices, version 56+);
Chromium on HTC Vive, Oculus, Android;
Chromium WebVR Build ( https://webvr.info/get-chrome/ );
Samsung Internet (Samsung GR VR);
Oculus Carmel (this is a WebVR browser, also available in the Samsung GR VR);
iOS Chrome (with polyfiles at the time of this writing);
iOS Safari (with polyfiles at the time of writing ...);
Chrome (with polyfiles for the moment ...);
Firefox (with polyfiles on ...).

A complete list with compatibility tables can be found here .

Do not forget to enable WebVR API support.

Polyphils and auxiliary libraries

If your device does not support WebVR API, then you can use a polyfill that can be connected to the page, or use a special browser extension. Polifil link: github.com/googlevr/webvr-polyfill

WebVR API Emulation for Chrome

You can put on the link: chrome.google.com/webstore/detail/webvr-api-emulation/gbdnpaebafagioggnhkacnaaahpiefil

About this plugin is to say a few words separately. It not only adds WebVR API emulation, but also allows you to do various manipulations and integrates into DevTools.

WebVR-UI

If your device is not a VR device (and the browser in the smartphone is just a browser), then you can recreate the VR interface to switch to headset mode using the webvr-ui library. With this library, you can make a beautiful UI interface with buttons to switch to VR mode. Link to the project: github.com/googlevr/webvr-ui

Adding a few lines of code:

 var renderer = new THREE.WebGLRenderer(); var options = {}; var enterVR = new webvrui.EnterVRButton(renderer.domElement, options); document.body.appendChild(enterVR.domElement);

This library is compatible with the A-Frame framework, which is very cool. You only need to add an attribute to your code:

 <a-scene webvr-ui> ... </a-scene>

And you will have a convenient VR interface. If WebVR is available, it will be offered to switch to VR mode, otherwise it will be offered to try to launch your web application in demo mode without VR mode.

This library is highly customizable, so you can make your custom buttons.

How can we build a world?

To integrate into the world of VR, you need to learn how to work with 3D graphics. This seems like a challenge if you learn WebGL from scratch. But the easiest way to get ready-made libraries, such as Threejs, D3.js or specialized frameworks for working with VR (such as A-Frame from the Mozilla team). There is even a preview of the release of React VR for React fans and everything that comes out of the bowels of FB. But this is a topic for a separate article.

Type VR world

Using the A-Frame framework, you can create a VR world with minimal effort. A-Frame is an HTML framework for creating web applications and virtual reality sites. If you look at the result in the browser, it looks like a web page that represents a 3D image with the ability to navigate and interact with objects. The whole picture is drawn using WebGL, and the main task of the framework is to provide a simple tool that would allow you to create a three-dimensional world in the usual way for front-tenders, similar to HTML markup, in a way. The framework itself is based on Threejs and is a higher level superstructure.

The A-Frame framework creates a 3D scene through a set of geometric primitives. You can also add more complex components and even add your own, if you already lack basic ones. For the objects of the scene are available typical geometric properties, such as: location, rotation, scaling, in addition, you can describe the location of cameras and light sources.

A-Frame has a good inspector that allows you to debug the 3D world.

Many good examples can be found here .

They are easy to understand and, having studied them, you can very quickly make your first “Hello VR world”. A-Frame can also be used for AR development. If you delve into the A-Frame, then again you can talk a whole separate article (which will appear if there is interest in this subject).

React VR

ReactVR is a framework based on Reactjs. If there is interest and the post has a good rating, I will make an article about it separately ... For now, just leave a link:

developer.oculus.com/blog/introducing-the-react-vr-pre-release

If the article scores a good rating, thus showing that this topic is interesting to readers, then I will make an article about it separately.

Can we do without WebVR?

As mentioned above, the WebVR API itself is needed only for working with VR devices, or rather, for obtaining characteristics about displays and for rendering pictures. But if we are talking about Mobile VR, then physically we work with the 1m screen. But we can emulate a stereo pair, and the WebVR API can help us to abstract and encapsulate the work with the screen in such a way that we will work with two physical displays. In this case, our code will be cross-platform, and we will be able to run our WebVR application on Oculus in the Carmel browser, for example.

An example of a 3D world on Threejs using the WebVR API

Here I will show an example of working with a boilerplate, thanks to which you can do WebVR projects that will be created on all browsers. Even in iOS.

Ready demo: webvr.majorov.su/sample1

 <!DOCTYPE html> <html lang="en"> <head> <title>Web VR boilerplate demo</title> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0, shrink-to-fit=no"> <meta name="mobile-web-app-capable" content="yes"> <meta name="apple-mobile-web-app-capable" content="yes"/> <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent"/> <style> body { width: 100%; height: 100%; background-color: #000; color: #fff; margin: 0px; padding: 0; overflow: hidden; } /* Position the button on the bottom of the page. */ #ui { position: absolute; bottom: 10px; left: 50%; transform: translate(-50%, -50%); text-align: center; font-family: 'Karla', sans-serif; z-index: 1; } a { color: white; } </style> </head> <body> <div id="ui"> <div id="vr-button"></div> <a id="magic-window" href="#">Try it without a headset</a> </div> </body> <script>WebVRConfig = { BUFFER_SCALE: 0.5 }</script> <script src="node_modules/es6-promise/dist/es6-promise.min.js"></script> <script src="node_modules/three/build/three.min.js"></script> <script src="node_modules/three/examples/js/controls/VRControls.js"></script> <script src="node_modules/three/examples/js/effects/VREffect.js"></script> <script src="node_modules/webvr-polyfill/build/webvr-polyfill.min.js"></script> <script src="node_modules/webvr-ui/build/webvr-ui.min.js"></script> <script> var lastRenderTime = 0; //   VRDisplay. var vrDisplay; var boxSize = 5; //   THREE.Objects. var scene; var cube; var controls; var effect; var camera; // VR UI  var vrButton; function onLoad() { //  three.js WebGL . var renderer = new THREE.WebGLRenderer({antialias: true}); renderer.setPixelRatio(window.devicePixelRatio); //       DOM document.body.appendChild(renderer.domElement); //   scene = new THREE.Scene(); //   var aspect = window.innerWidth / window.innerHeight; camera = new THREE.PerspectiveCamera(75, aspect, 0.1, 10000); controls = new THREE.VRControls(camera); controls.standing = true; camera.position.y = controls.userHeight; //  VR stereo  effect = new THREE.VREffect(renderer); effect.setSize(window.innerWidth, window.innerHeight); var loader = new THREE.TextureLoader(); loader.load('img/box.png', onTextureLoaded); //  3D  var geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5); var textureLoader = new THREE.TextureLoader(); var texture0 = textureLoader.load('img/1.jpg'); var texture1 = textureLoader.load('img/1.jpg'); var texture2 = textureLoader.load('img/0.jpg'); var texture3 = textureLoader.load('img/0.jpg'); var texture4 = textureLoader.load('img/2.jpg'); var texture5 = textureLoader.load('img/0.jpg'); var materials = [ new THREE.MeshBasicMaterial({map: texture0}), new THREE.MeshBasicMaterial({map: texture1}), new THREE.MeshBasicMaterial({map: texture2}), new THREE.MeshBasicMaterial({map: texture3}), new THREE.MeshBasicMaterial({map: texture4}), new THREE.MeshBasicMaterial({map: texture5}) ]; var material = new THREE.MultiMaterial(materials); cube = new THREE.Mesh(geometry, material); cube.position.set(0, controls.userHeight, -1); scene.add(cube); window.addEventListener('resize', onResize, true); window.addEventListener('vrdisplaypresentchange', onResize, true); // Initialize the WebVR UI. var uiOptions = { color: 'black', background: 'white', corners: 'square' }; vrButton = new webvrui.EnterVRButton(renderer.domElement, uiOptions); vrButton.on('exit', function () { camera.quaternion.set(0, 0, 0, 1); camera.position.set(0, controls.userHeight, 0); }); vrButton.on('hide', function () { document.getElementById('ui').style.display = 'none'; }); vrButton.on('show', function () { document.getElementById('ui').style.display = 'inherit'; }); document.getElementById('vr-button').appendChild(vrButton.domElement); document.getElementById('magic-window').addEventListener('click', function () { vrButton.requestEnterFullscreen(); }); } function onTextureLoaded(texture) { texture.wrapS = THREE.RepeatWrapping; texture.wrapT = THREE.RepeatWrapping; texture.repeat.set(boxSize, boxSize); var geometry = new THREE.BoxGeometry(boxSize, boxSize, boxSize); var material = new THREE.MeshBasicMaterial({ map: texture, color: 0x01BE00, side: THREE.BackSide }); skybox = new THREE.Mesh(geometry, material); skybox.position.y = boxSize / 2; scene.add(skybox); setupStage(); } // Request animation frame loop function function animate(timestamp) { var delta = Math.min(timestamp - lastRenderTime, 500); lastRenderTime = timestamp; cube.rotation.y += delta * 0.0006; if (vrButton.isPresenting()) controls.update(); effect.render(scene, camera); vrDisplay.requestAnimationFrame(animate); } function onResize(e) { effect.setSize(window.innerWidth, window.innerHeight); camera.aspect = window.innerWidth / window.innerHeight; camera.updateProjectionMatrix(); } function setupStage() { navigator.getVRDisplays().then(function (displays) { if (displays.length > 0) { vrDisplay = displays[0]; if (vrDisplay.stageParameters) { setStageDimensions(vrDisplay.stageParameters); } vrDisplay.requestAnimationFrame(animate); } }); } function setStageDimensions(stage) { var material = skybox.material; scene.remove(skybox); var geometry = new THREE.BoxGeometry(stage.sizeX, boxSize, stage.sizeZ); skybox = new THREE.Mesh(geometry, material); skybox.position.y = boxSize / 2; scene.add(skybox); cube.position.set(0, controls.userHeight, 0); } window.addEventListener('load', onLoad); </script> </html>

In the following articles we can understand in more detail the details of the development for WebVR. Link to github with a boilerplate: github.com/borismus/webvr-boilerplate

And what about Web AR?

Web AR (Augmented Reality) - it is also possible to create in the browser. But the “WebAR API” does not exist, it is simply a designation of augmented reality implemented on web technologies.

Technically, all the work is exactly the same as working with WebVR, but in addition you get a video stream from a webcam using WebRTC. Additionally, write the logic of frame-by-frame processing to find the necessary objects. And further, as in the case of WebVR, you are already drawing a 3D scene against the background of a video stream. And I must say that AR does not necessarily imply the presence of a headset. Recall “Pockemon GO!” Is an AR project, but without a VR helmet. It follows that to create an AR project, it is not necessary to have a VR helmet. But at the same time, VR and AR concepts can intersect in some parameters and technologies.

What with support?

Here everything rests, first of all, in support of WebRTC. Therefore, we can say that Web AR can be implemented on all Android devices. Not in iOS, but if you really want, then ...

How to deal with iOS devices?

If we are talking about WebVR, then in iOS devices everything can be implemented through polifila (or you can do without WebVR API, describing everything on your own and implementing action tracking via an accelerometer and other sensors, beat the picture and emulate two displays). If we talk about AR, everything is bad, as there is no support for WebRTC. But there is such a project as Argonjs. This is a project that consists of a framework based on A-Frame, and, attention, a special browser.

Argon 4 by Georgia Tech

→ Link to project: argonjs.io
→ Demo: argonjs.io/samples

The essence of the browser is simple: there are two layers. One layer is the Webkit engine, the second layer (substrate) is the output of the video stream from the camera. The browser has its own API, something similar to the work of the very first versions of PhoneGap (if someone tried at the dawn of development of this project, in 2008, then it was a special iPhone browser with advanced JS API).

At the same time, Argonjs can also be used for VR development for iOS. More precisely for running MVP. If you still need to develop a cross-platform application for VR / AR on JS, then you can look towards React Native along with React VR or try to pack everything into PhoneGap.

Important! In the AppStore there are several versions of the browser. At the moment, you need to download version 4, not lower, otherwise the framework will not start:

itunes.apple.com/us/app/argon4/id1089308600?mt=8

And what about gamepads?

This is an additional topic to learn. We understand with Bluetooth API and Gamepad API. They are in browsers and are supported both on the desktop and on mobile devices (again, there are nuances and a list of favorites among browsers). We study the documentation and learn to work with them. About gamepads and interaction with the VR world is the topic of a separate article (or even several), not to tell in two words.

Use cases and application

If we talk about Tutu.ru , then we already have ideas for use cases for using VR / AR in practice. For example, these are 3D galleries for the Tour project. Galleries 360 with a slight revision can be adapted to VR helmets, and everyone who has a VR Box at home could “explore” the future hotel and beach with a presence effect. The same technique can be applied to 3D galleries for the project of the Train , where you can see the car from the inside and choose a place directly from the VR helmet in Sapsan. In the same way, in the future, could be done for intercity buses. And for the excursion project, which we also have, it would be possible to make a demo preview of the future excursion.

Another case is the entertainment of tourists in the offline office . While the tourist is waiting for his turn to be a tour manager, instead of magazines and brochures one could put Samsung GR VR helmets and, again, make a travel catalog, which could help choose the right package. If we develop the theme of AR, then in the offline office one could place information stands and various priks-prikolyukh, for the entertainment of customers. If the client can skip the brochure, because the reaction to advertising waste paper is not always positive, then the client can enjoy and be pleased with the same advertisement shown through the prism of new technologies.

In general, there is a flight for fantasy. I told them the cases in the context of our company, and they are not all listed here. Speaking globally, you can think of user cases with a car and a small cart :)