Geekly Articles each Day
📜 ⬆️ ⬇️

Mobile VR development with Oculus and Gear VR


This very long post is dedicated to developing a mobile VR (untethered VR), as well as some design issues and issues that need to beware during the development process, as well as performance optimization, testing and publishing in the Oculus store. But before we begin, let's think about why we need such a development.

Currently, there are two main directions of VR-development - tethered and untethered, that is, based on computers and based on mobile devices. Computer tethered VR uses powerful desktop graphics cards to create images in high-end VR devices like the Oculus Rift and the HTC Vive. For untethered VR mobile, use a user's mobile device inserted into a plastic, cardboard or foam VR device; it uses only the computing resources of the user device, the characteristics of which can vary over a wide range.

Reflections on mobile VR


Today, mobile VR is the cheapest way for consumers to immerse themselves in virtual reality; For her, there are many supported mobile devices and glasses with a low price. Despite some problems (we will discuss them below), modern and future models of mobile devices are powerful enough to provide convincing VR sensations that are comfortable for most users (up to a decent 60 Hz on Samsung Gear VR). More than 5 million Google Cardboard points are sold, and Samsung claims a million users in the Gear VR ecosystem.

The range of glasses is very wide, from cheap Google Cardboard v2 (often distributed for free at various events) and more robust plastic variations, as well as foam versions, such as Merge VR, to Samsung Gear VR and a whole sea of ​​other glasses connected to mobile devices. to be released soon. A wide range of devices and variations is good, but for VR development, ideally, you need an accurate set of studied hardware limitations and capabilities to provide the end user with reliable, smooth and convenient gameplay. Therefore, we will focus on the best devices in our class today: Samsung Gear VR and Google Daydream VR (to be released towards the end of 2016, but you can start development right now, study the previous post here ).
')
Due to the low cost of entry for end users and consumers, the mobile VR market is growing rapidly, hundreds of new apps appear monthly. In 2016, thanks to the release of the Oculus Rift and the HTC Viveo, as well as the early sales of Sony PlayStation VR, the availability of virtual reality has finally become known to the general consumer; VR appears on many public events and shows.

Despite the fact that Cardboard is supported only on Android (iOS is mutually incompatible with it), developers can conveniently publish their applications to the Play Store; there we already see a bunch of cheap, low-quality VR-applications that do not provide any pleasure from the gameplay. Therefore, Gear VR and Daydream online stores (after its release) will carry out quality control.

Of course, this creates additional barriers to the publication of applications, but the end users at the same time receive a guarantee of a certain level of quality, which will accelerate the spread of VR: thanks to high quality, users will play their games more often and try to acquire new ones. In the short term, quality control is a headache for developers, but overall it’s better for everyone because it helps VR get into the mainstream market.

Restrictions


Mobile VR has its limitations, which must be considered before choosing a supported platform. First, as the name implies, mobile VR content is provided by mobile devices rather than powerful computers. Although the new devices have much more computational resources, there are still limitations on the depth and complexity of the scene rendered in 3D (if you choose to go this way). Of course, this does not mean that the devices are completely ineffective: for the scene generated in 3D to be convincing for users, photorealism is not required of it. Low-poly worlds generated on mobile devices are quite interesting and effective in VR, while they provide high performance.

Another aspect is that, despite independence from the computer and providing the user with freedom of movement without cables, it is impossible to track the movement of glasses in space: it does not matter how the user moves, the camera position is updated only in three directions. This means that the multitude of full VR sensations, obtained with tethered VR (squats, approaching the characters from any angle, free movement within the room), are not available for mobile VR. However, equipment is being developed that should sooner or later solve this problem.

So, the mobile VR has less computing power and there it is impossible to track the position of the head; it is clear why applications of panoramic viewing of photos and videos dominate and are well supported. However, they lack any means of real interaction, the user can only look at something, so some do not consider this “correct” VR. Others also believe that, thanks to improvements in mobile VR and increased computing power, developers should target full, high-quality VR, because the gap will only narrow, and if today you are developing applications for mobile VR, they will soon become obsolete within a few months or years due with the rapid development of equipment.

Preparation for development


Publishing apps to the Google Play Store is easier, and this is a great place to start getting started, if you are satisfied with the release of free content, but I will mainly focus on developing content for the store that controls the quality of content offered by Oculus for Samsung Gear VR; All recommendations will also be valid for the Google Daydream VR glasses ecosystem, which will be released this year.

Development environment, SDK, and how to become an Oculus developer


To create a 3D VR application, you need a 3D development environment that supports the necessary software development kits (SDKs) for VR. In our case, this is the Oculus Mobile SDK. The best 3D engine for this is Unity, which has free and paid versions. The free version has certain limitations and requirements for use, and the paid version already contains everything needed to publish a mobile VR for Android devices (for example, Samsung Gear VR and Google Daydream VR) and iOS (which in fact does not yet have strong support for VR, although VR apps for iPhone and supported by Google Cardboard); You do not have to pay for these features separately, as in older versions.

After installing Unity, you need to download the Oculus Mobile SDK for Unity and become an Oculus developer. To do this, you need to go through a simple registration in order to create an Oculus ID to access the website and highlight an area in which you will then record your applications for verification before publishing. Mobile SDK has gone through many iterations, making it ready for work and has very detailed documentation. Together with the Mobile SDK, you can also download the Audio SDK for optimal use of the Oculus sound positioning features, which add immersive sensations for users of VR applications. It is also now convenient and well documented.

After setting up your Oculus ID, you can also register on the Oculus forums (as well as Unity) to search for answers to any questions and solutions to emerging problems, as well as to become part of the growing community of VR developers.

Equipment


We looked at software, but for professional development and building quality applications, you also need hardware so that you can build, test, and run your applications locally before being sent to the store. If money is not a problem, try to find as many supported Samsung mobile devices as possible to test the application and test performance. If you don’t have enough money, buy the “lowest common denominator” - the Samsung S6, to make sure that the application works at least with the minimum supported requirements. The availability of testing equipment also means that you can give users a chance to test the application before release and get feedback. Usability and user friendliness are key elements of VR development, it is very important that in your VR applications the user does not “rock”! But this is a topic for another post ...

Make sure the mobile device is in developer mode and it allows the installation of applications from unknown sources. To enable developer mode on Android, go to Settings> About device> Build number ( Settings> About device> Build number ) and tap Build number seven times. Then go to Settings> Application Manager ( Settings> Application Manager ), select Gear VR Service , and then Manage Storage . Tap the VR Service Version several times until the Developer Mode switch appears. Switch it to enable Oculus Gear VR Developer Mode . And finally, we need to allow the device to install and run applications from unknown sources, because when testing the application it is not downloaded from the Oculus store or Google Play Stores. To do this, go to Settings> Security ( Settings> Security ) and turn on the Unknown sources option, and then select OK in the dialog box. After that, the mobile device will be able to run the developed applications for testing and testing demo. Note. If the Gear VR Service option is not available, you need to insert the device into the Gear VR glasses, and then complete the installation and configuration process of the Oculus Gear VR software, drivers, and applications - yes, this means that you have to buy glasses!

The final element to create a connection between the hardware and software is to get the OSIG mobile device to insert it into Samsung test assemblies so that they work in the Oculus Gear VR system without being a proven application. To do this, you must first download the Device ID app from the Google Play Store, and then launch it to get the Device ID of your devices. When you receive them, go to the Oculus OSIG Oculus Generator website and enter the Device ID to get a unique OSIG file. After downloading, paste it into the application package in Unity to ensure that the application builds are launched on the mobile device.

With hardware and software, what about design and development?


Of course, not every application that you developed should be released and available for public use, so a good approach (especially in the case of VR) will be to create the first series of prototypes and implement simple ideas before moving on to a full development cycle polished to the brilliance. Google has created an enormous amount of interactive prototypes for Daydream VR, simple and effectively allowing developers to understand the features that will be available with the new equipment and input controller. If you are new to VR development, you will also need to study them to understand what works and — most importantly — does not work in VR, as well as to understand the current limitations of mobile VR. If your application is slow or uncomfortable to use, it will not be published.

Do not complicate


Because of the limited input capabilities in a mobile VR, simple ideas and interactions work well. Of course, there are bluetooth controllers for Android, and they are supported by Samsung Gear VR, and it is also stated that Google Daydream VR will come with a compatible controller, but most users do not yet have bluetooth gamepads. Therefore, if you are developing an application (usually a game) that only works with a bluetooth gamepad and does not support the touchpad on the sides of the glasses, then you significantly reduce the size of the potential market for the audience of buyers.

The touchpad on the side of the Gear VR (v1) is embossed and resembles the D-pad game controller, so it’s harder to do diagonal scrolling (swipes) and interactions, but for new VR users it’s easier to scroll forward, backward, up and down with a protrusion marking the center of the touch area of ​​the button. The pre-order version of the Gear VR 2, which will be released later this year, has returned to the design of the flat touchpad, as in the earlier versions of Innovator Editions. This is a good solution in terms of design, because this way you can track more complex movements of the user's fingers.

The main drawback of the touchpad on the side of the device is that new VR users usually grab the glasses, holding them with both hands and getting used to the VR sensations. This often leads to a random exit or inclusion of a pause in the application, depending on the design, which is confusing and makes it difficult to show demo versions, because unlike tethered VR, you cannot see what users see if what is happening in the phone doesn’t displayed on any display.

Keep in mind that for most users trying your application, this can be the first VR experience: if the input method is simple, they will learn technology much faster that can be amazing for them by themselves. Fortunately, the popularity of VR is growing, more and more people have access to VR, so this factor will need to be considered less and less; We hope that next year we will not have to take care of it at all.

Stable performance


In order for the application to be approved and user-friendly, it must consistently deliver 60 frames per second (FPS) to the Samsung Gear VR. This is necessary because such a bar is recognized as the minimum comfortable VR frame rate for most users. Any reduction in the frame rate, even for a short time, can lead to a “motion sickness” of users, because the virtual world will start moving in spurts and twitching, trying to match the movements of the head.

This can be a daunting task if you are not used to using 3D engine optimization or simplifying geometry. In any frame of your VR scene, 50,000 polygons should be displayed (maximum 100,000), so you need to think things through, calculate in advance and use Unity's tricks to ensure a good picture, not forgetting about stability.

Fortunately, the latest version of Unity 5.4 supports rendering in a single pass, so you can get the same result by spending less hardware resources; the VR elements will be taken care of by the engine - it does not render everything twice, but renders a frame from slightly different angles to create the scope of each eye and the desired 3D depth effect.

John Carmack, one of the authors of DOOM, now works at Oculus and spends a lot of time on tools and development for mobile VR. Thanks to this, Gear VR has long been supported by asynchronous timewarp, a technique used by the SDK to smooth out dropped frames and allow developers to get rid of image jerks on mobile equipment. But this does not mean that you can use it as a “crutch” and not to optimize your code! It has its limitations and can not always save you and users.

Oculus claims that the following limitations should be considered when developing a mobile VR for Gear VR glasses:


Note. All other Android APIs and SDKs (such as Google Cardboard) usually do not provide access to direct management of the central and graphics processors of a mobile device, this feature is provided only by Oculus for selected Samsung device models thanks to partner assistance in creating Gear VR and mobile SDKs.

General recommendations and design tips


The time has come to review various helpful tips and tricks related to general VR recommendations and design in order to provide beginners and advanced users with the great VR experience.

General tips



VR



User interface




If you’ve read this far, when we’ve already reviewed why it’s worth creating mobile VR applications for Samsung Gear VR (and preparing a bit for Google Daydream VR), how to set up a development environment and general VR design guidelines, and still with us, then you should be interesting enough! Therefore, now we will assume that you have at least a basic knowledge and understanding of creating 3D assets and application development, because I will move on to more specific terminology in order to examine in detail the key aspects of design and development. So remember, I warned you ...

Performance optimization


Optimizing the performance of a mobile VR application is a key factor guaranteeing user comfort (for now, let's forget about horror games) and passing the test as a process of publishing in a store.

There are several aspects of performance: overall application performance, 3D optimization and battery consumption. All of them play an important role in ensuring that users play the game as long as possible, leave good reviews and tell friends and acquaintances about it.


Testing


The key to creating a quality application is regular testing with iterations between each session, including suggestions and improvements in gameplay, interface, development, and design, instead of postponing all changes to the last minute, when you think that everything is 100% complete. Thus, you can constantly make small improvements that generally require less effort than the sudden detection of a major flaw in the design, found too late and requiring a tremendous amount of effort for processing and correction.

As a developer, you are too immersed in the process and you may overlook the problems and bugs of the application, so user testing, which starts in the early stages and continues throughout the development process, is critical so that you do not miss something obvious that the user will immediately notice. . However, there are tests that you can and must perform before showing the application to others.

The main types of testing that you will conduct are related to functionality and performance to ensure a basic level of proper operation of the application, sufficient for comfortable use by players. You can write unit tests to test various aspects of the functionality, but you still have to do manual testing and find the problems yourself.

If you decide to manage the development process using the Agile methodology, you can create test cases based on your own feelings and user stories so that the application runs and contains all the necessary functions. Otherwise, you will need to think about test cases that can effectively test and cover all possible conditions and actions: not only the expected behavior and actions of the user, but also what the user can do to disrupt the process and potential “getting stuck” in the game (not Check the required box or do not score enough points to go to the next level without the retry option).

Testing VR for functionality is more difficult than testing standard, flatapplications, because it’s better to perform it in a VR-glasses device, but this means that you will not be able to quickly switch to a spreadsheet or notepad to quickly write down the problems found. Therefore, it is recommended to perform paired testing, in which one tester is in the application and performs all the tests, while the other writes the comments that are spoken aloud first.

But until you get to this stage, you can run the application directly in the Unity editor to check functionality and performance without building and installing the application on a mobile device. As described in the section on performance optimization, the profiler, Frame Debugger and Scene View in Unity allow you to perform initial performance testing, and the editor itself informs the developer about limit exceptions and errors in the code.

User Testing


User testing requires more preparation and time to guarantee high-quality reviews about the application, not about the technology. As I mentioned in my post earlier, about 9 out of 10 people have not tried VR yet, so using them as novice testers requires management for preliminary preparation.

In the process of organizing a user test session, each tester should have time to become familiar with VR and “acclimatize”, after which you can begin to try testing the application. If testers are new to VR, this will give them a chance not only to be overwhelmed by the technology and sensations of immersion in another world, but also to be useful by providing feedback directly about your VR application. When they know what VR can do and how it works at a basic level, they will be ready to test your application, understanding more clearly how it should work and what feelings and sensations should evoke. Good examples of familiarity with applications for Gear VR include Samsung Introduction to Virtual Reality (free) and Welcome to Virtual Reality by SliceVR (paid).

Prepare a set of questions to ask users about your application's testing session, to get helpful feedback and information about how they felt, how easy it was to understand what to do, where they had problems or difficulties, and about the aspects that were for uncomfortable (in terms of performance, not content, especially in the case of horror).

Remember that content displayed in a mobile VR device is more difficult to display on a separate monitor, and therefore you will not be able to see in real time what they are looking at (and what they are pointing to in the air). Make a set of prints of important screens and application menus so that after a test session, users can look at them and describe the different screens and panels that they saw (but they will not necessarily call them the same names that you gave to these elements).

If your budget is large enough, you can use the services of companies offering VR testing services to reduce your own labor and time costs. Testronic Labs offers a VR pairing test for functionality and compatibility, and Player Research is a leader in user research and user testing; this company creates and provides developers with post-test detailed reports that can be used as part of the service.

So, your application is already stable at 60 FPS and does not contain errors (as far as possible for software), it was tested and found to be comfortable and easy to use for the target audience of end users. It's time to send the app to the Oculus Store and get ready for the release!

Send to the store


The process of sending to the store


To be able to sell an app for Gear VR in the Oculus Store, the app needs to be tested by the Oculus Store team for comfort, performance, and overall compliance, after which the “green light” is given to release the app.

This is a fairly straightforward process, but it requires a little effort on your part for internal tuning. The amount of labor required for customization depends on the functions used in the application (for example, the availability of in-game purchases (IAP), the organization of matches for several users, achievements, leader boards, etc.). Many items require an internal API and ID configuration, so you need to go back to the application project in Unity to make sure that you use the correct values ​​to unlock every achievement, in-game purchase, etc.

You should have already set up your Oculus ID, but if you haven't done so already, go to oculus.com. In the Developers> Dashboard section, you need to select Create new organization in order to be able to create an application profile. Make sure that important information about your organization is entered correctly (address, financial information, etc.) so that you can get the equipment, and, more importantly, monthly payments for the purchases of your application.

The basics


After setting up the organization, you can proceed to setting up the application: go to My Apps> Create New App and enter the necessary information.

IMPORTANT!Currently, the ability to completely delete the application after it is created is missing, so in the process of creating, make sure that all the information is correct - you can go back and change the information about the application at any time, but if you, like me, like a blank panel, do everything right right away

The first step is to choose a platform: we will start a mobile VR application, so select Gear VR and enter the full name of the application to create the first entry.

After creating the first profile of the application with the name and platform, Oculus generates a unique application ID that you need to use in the Unity project to initialize various Oculus APIs, especially those related to in-game purchases and checks for the license for the final version.

Also, after entering the financial information, you can create any IAP tokens and IDs that you can then call in the Unity project for appropriate actions via Edit Details> Platform> IAP .

Application Store Information


The main information of your application in the Oculus Store is in Edit Details> Submission Info . Here you can enter a full and brief description, select a genre, features, supported peripherals, age limits and cost.

Some of these elements you choose yourself, some require the conclusion of third-party organizations (for example, age restrictions); regarding the cost of Oculus Store employees will contact you and agree on the appropriate amount. Do not forget that by default all applications are considered free, and if you want to sell them, you need to change the cost before shipping!

Another time-consuming element, just like when sent to any store, is artistic assets that need to be created for the application to have its own record. Images of various shapes and sizes are required, depending on how and where the record will be displayed in the store, but the instructions are clear and easy to use (for example, about the need and location of the logo so that the included dynamic sales banners do not close it).

There is one interesting asset that can be added to applications for Gear VR (and which is not yet supported by applications for Rift): this is an image of a cubic texture, so that potential customers can see the panoramic view of your application in Gear VR.

Assembly


Of course, the information is very important, but you also need to download the assembly of the application for review and for downloading after purchase. Before sending, you need to run the Oculus Submission Validator tool for your APK file and make sure that you have the following:


The build management section allows you to download builds for various channels: Alpha, Beta, Release Candidate and Live. It should be noted that many journalists have access to the Release Candidate channel, so if you have not worked on PR or marketing applications before the release, then be prepared for the fact that they may accidentally stumble upon it and issue an article review without warning. So, before uploading an assembly to this channel, it is better to contact them to eliminate all problems if the application is not 100% ready for release!

The application is ready to be sent.


If you decide that all data has been entered, you can send information in the Submission Info> Submit section , which has a convenient list with the current status of each desired section. When a beautiful row of green flags finally appears, you will have the last opportunity to check the entire list and all the information before clicking on the [SUBMIT FOR REVIEW] button. After the application is submitted, Oculus Store Gear VR employees will check the application information, test it and contact you to report any proposed changes, after which the application will be considered ready for release.

THAT'S ALL! You have gone all the way - great work, success in mobile VR development, share links to applications created by you in the comments.

tl;dr


  1. Samsung S6 S7: http://www.samsung.com/uk/consumer/mobile-devices/smartphones/
  2. Samsung Gear VR HMD: http://www.samsung.com/uk/consumer/mobile-devices/wearables/gear/SM-R322NZWABTU?catnm=Gears+%26+Watches&catid=4340
  3. Unity: https://store.unity.com/
  4. Oculus Developer Centre, Oculus ID: https://developer.oculus.com/
  5. SDK Oculus: https://developer.oculus.com/downloads/mobile/1.0.3/Oculus_Mobile_SDK/
  6. SDK Oculus: https://developer.oculus.com/downloads/audio/1.0.4/Oculus_Audio_SDK_Plugins/
  7. Android Device ID Google Play: https://play.google.com/store/apps/details?id=com.redphx.deviceid&hl=en_GB
  8. Device ID OSIG Oculus: https://developer.oculus.com/osig/
  9. OSIG Unity
  10. VR!


» VR Unity
» Oculus VR
» Oculus Mobile SDK
» Unity
» Oculus


» Oculus Blog - Squeezing Performance Out Your Gear VR Title pt.1
» Oculus Blog - Squeezing Performance Out Your Gear VR Title pt.2
» Oculus Mobile SDK - Testing & Troubleshooting
» Unity - Optimization for VR
» John Carmack's Post Smoothing (anti -aliasing) on ​​Facebook

Useful links for sending applications


» Oculus Developer Dashboard
» Oculus Store - Instructions for Art Assets for Gear VR
» Oculus Submission Validator
» Review of Oculus application publishing

Source: https://habr.com/ru/post/310792/

More articles:


All Articles