"The threshold of entry is reduced to the knowledge of the basics of computer graphics": the developer Alexander Korshak about mobile VR

Back in May 2016, Google announced the Daydream VR platform loudly, but only now it becomes really relevant: other smartphones began to receive the Daydream-ready label after the Pixel flagship. Does this mean that it is time for Android developers to deal with VR? Does it make sense to devote your time to this if you have no relation to the gaming industry? What do you encounter in practice in mobile VR development?

Such questions can best be answered by a person with relevant experience, and Alexander Korshak has this experience: first, he made a VR application from personal interest, then at MERA developed a mobile video editor with support for 360-degree video, and now participates in a startup, associated with spherical video, Internet and TV-broadcasts. As a result, he managed to work with the same Daydream, with the Cardboard that preceded him, and with the Gear VR - and we asked him about everything at once.

')
- The first question that arises in connection with mobile VR development: there are a whole bunch of devices for smartphones (Daydream helmet, Cardboard, Gear VR, helmets of companies like Homido), which of these should be developed separately?

- In fact, everything is quite simple. In mobile VR, there are two giants, two main platforms. One is Oculus, Samsung's Gear VR from Samsung works on this platform. Everything else is the former Cardboard, now Daydream, i.e. Google’s consolidated VR platform. The Cardboard and Daydream APIs are almost indistinguishable; the Google VR SDK is used for development for both platforms. Therefore, with a large variety of headsets, if you look at the platform part, everything is not so bad.

In addition, as in the case of mobile development, there is the possibility of cross-platform development, but not for conditional iOS and Android, but for Oculus, Cardboard and Daydream. Unity immediately comes to mind, but there are other solutions.

- At the same time, Daydream has a controller compared to Cardboard. Did it all change that much?

- Well, from the user’s side, of course, that changes everything radically. The interactive, which used to be reduced to simply turning your head in VR space, is now achieved through hand movements, buttons, touchpad - in short, everything we are used to over the years of using smartphones. This is very cool, users have much more degrees of freedom in the VR world, and developers have more opportunities to implement their ideas.

But technically, the controller is just a set of sensors, a pair of buttons and a touchpad. A developer with a controller API can be handled without much difficulty. The application written for Cardboard will easily run on Daydream. The application written for Daydream will run on Cardboard, but without the support of the controller, its API will simply not be called up and working.

- But at the same time, probably, some Daydream-applications without a controller will be meaningless?

- Yes, there is such an opportunity; there's nothing to be done about it. Although you can always add support for a third-party gaming controller and use it instead of the “deidrim” one. In this case, we get the same thing, only in a different way. But, of course, the question is whether users want to connect a third-party controller.

- And on iOS now VR is in what condition?

- In a very rudimentary. About Cardboard everything is clear, but Cardboard is no longer quoted. And in the case of Daydream, innovation consists not only in adding a controller, but also in updating the entire Android specifically for the platform (I talked about this in more detail at Droidcon Moscow). That is, the Google developers have added additional optimization tools to their system that work automatically and allow you to achieve the optimal quality and performance of Daydream applications if they are running on Android 7.x. Since iOS is not owned by Google, they could not change it - and Apple has not yet introduced any of their own VR solutions.

- In connection with mobile VR, they talk about the problem of strong heating of devices - what does your experience say about this?

- The problem is very acute and urgent, yes. The only means by which now any platform avoids overheating of the device is to force the application to work in VR mode. The system will show a message stating that the device requires cooling, and further operation of the application is temporarily impossible.

You clearly feel this problem when developing. During the day, you have to spend a lot of debugging code, the devices are very active and often overheat.

How to deal with the problem, I do not know. Experience of the gaming industry tells us that gaming laptops are not those that just have very good hardware installed, but those that have a good cooling system along with this hardware. And in the case of a mobile VR, it is very hard to come up with some kind of cool cooling system that can pull out the work with high graphics settings, while maintaining the mobility of the device.

But a lot depends on what exactly to show in VR. In the case of games, of course, the problem is interesting, but in the 360-degree video that I do, everything is better. The rule is simple: the less we draw on the screen and the less often, the longer the device heats up. In the case of games, these are millions of particles that need to be rendered at least 60 FPS, which is very difficult for the CPU and GPU devices. In the case of video, simply rendering video frames and playing audio in parallel, which is much simpler.

- Heating is a “pitfall”: you don’t learn about it from beautiful presentations of VR platforms, you come across it in practice. What other non-obvious problems did you encounter on personal experience?

- It depends on which platform to talk about. In the case of Daydream, I remember how I tried to develop before the Daydream helmet and controller came on sale, using Android 7 Preview and a controller emulator. It turned out that the emulator with a very strange frequency returns the accelerometer and magnetometer data: there were large, unexplained delays in data transfer. Bad experience. With the controller, which is now sold, everything is much better, but periodically there is a problem with the sensors.



- Daydream was announced less than a year ago - how stable is it now felt, do you have to deal with problems of the form “still damp”?

- Problems "out of the blue" are almost not visible, although a year ago they were. For example, a year ago, the Google VR SDK refused to work on x64 architectures. There are fewer obvious problems, although, on the other hand, there are not obvious ones. The requirements of the industry are not fully formed, so much has to be implemented by yourself.

- The maturity of the platform depends on how easy it is to find the necessary information - is there already good documentation and ready to help the community? What's wrong with this mobile VR?

- Again, depending on which platform. Oculus I was unhappy a year ago: the documentation was constantly irrelevant from one version of the SDK to another, and on the Oculus forum it was hard enough to wait for answers from the developers and the community. In the case of the Daydream and Cardboard documentation, everything is great, the APIs are very nice and easy to understand, and therefore there have never been any special problems with them.

At the mention of VR, people usually think about applications that are completely tied to virtual reality. But you worked on a video editor, where only part of the functionality is connected with VR. Is this broad niche a “regular” application where VR is an optional addition?

- Yes. I like the example of one American real estate company, which added the ability to view apartments for sale in VR mode. For busy people, this is a great opportunity to find housing without wasting time ...

- In the video editor you have been working on, 360-degree video can be seen not only in VR, but also in “normal” mode, right? Do you have statistics on how to look more often?

- Yes that's right. According to the analytics, it was tracked that if a person opens a spherical video, then most often he views it in VR mode.

“Suppose the mobile developer, after this interview, wondered whether to add optional VR functionality to his application. The question is: wouldn’t it be inappropriate for him to deal with a new area for the sake of one feature? How high is the entry threshold?

- It seems to me that at the moment the threshold of entry basically comes down to knowing the minimum fundamentals of computer graphics, and no more. Again, if you start from Cardboard and Daydream, they have both the documentation and the API fairly simple and transparent for a developer who is just starting to engage in VR.

- Last question: do you have to deal with the fact that now something is not enough in the platforms of mobile VR, but in their next versions this may still appear?

- Yes, there are quite a few such moments from my bell tower of a developer who is involved in a 360-degree video. Here is an example. There are various 360-cameras on the market - Google Jump, Gear 360 and a number of similar ones. All of them shoot and transmit video in different formats and projections, which are not necessarily supported by the SDK for Cardboard, Daydream or GearVR. Accordingly, the problem arises of expanding the standard functionality of well-known frameworks, and even better to make it cross-platform.

At the next Mobius conference, I will talk a lot about video formats, customizing the playback of VR video, and so on ... All these questions arose just because the standard functionality of the Cardboard SDK or Oculus SDK was not enough for me.



Last year, at Mobius 2016, Alexander already talked about VR development, hitting the top reports of the conference. At that time, he had not worked with Gear VR yet, Daydream was just announced, and the report was based on the pet project experience. Now, having received a lot of work experience with a 360-degree video in the workplace, Alexander will share it at the conference in the new report “Spherical video in Android: 50 shades darker.” Mobius 2017 will be held in St. Petersburg on April 21-22.