Magic Leap released SDK. Let's see what's inside?

Magic Leap is a startup engaged in augmented reality devices. The company has already attracted $ 1.4 billion in investments from impressive investors, including Qualcomm, Google and Alibaba Group. In 2016, Forbes rated the Magic Leap at $ 4.5 billion. In this case, the public until this point Magic Leap showed only renders.

Magic Leap One

The ML1 includes the glasses themselves, which the manufacturer calls Lightwear, the Lightpack computer for calculations and the controller.

The manufacturer promises that the device will be quite easy and convenient for several hours of continuous use. Technology Digital Lightfield, implemented in this device, allows you to add objects of augmented reality, taking into account the environment.

Lumin os

The operating system in Magic Leap One uses Lumin OS. It is based on the Linux kernel. Some of the functionality was taken from Android, some were written by themselves, as it was necessary to ensure high performance for spatial computing tasks. The OS supports Wi-Fi, Bluetooth, multimedia and multitasking. Before using the microphone, camera and other devices related to personal information, the user will see a dialog box asking for access to them.

As a 3D-engine, you can use the Unreal Engine 4 or Unity, which are now standards in this area. You can also use the Lumin SDK C API to integrate Lumin OS capabilities into your engine.

Lumin sdk

The Lumin SDK includes a library that provides access to the ML1 device capabilities, a device simulator and a space simulator, allowing you to generate a room in which the device simulator will run.

Platform features

Headpose

Allows you to get the coordinates and direction of ML1 in the real world. For tracking used VIO (camera + accelerometer). If there are problems with cameras (low light, poor texture, overlap), the tracking of movement is turned off and only direction tracking works.
An interesting feature is that ML1 remembers the space around it. This allows you to restore the coordinates after losing them and moving to another part of the room. You can also save the coordinates of the objects of augmented reality in a certain place in the room and restore them when you start a new one.

Gestures

ML1 can recognize 8 gestures. As information, you will be given coordinates and probability, while the hands should not overlap each other. In the simulator you will receive information about gestures that are in the visibility of cameras, but not visible to the user. On a real device, you will receive information only about gestures that are in view of the user.

Audio

The SDK allows you to play "spatial" sound, taking into account the locations around the user. Without treatment, sounds will seem flat, insufficient for a sense of immersion. Soundfield Audio technology can be connected to projects on Unity, Unreal Engine 4 or through a plug-in to Wwise or FMOD. In future releases it will be possible to use calls to Lumin SDK directly.

Multimedia

ML1 supports audio and video playback using a media player. HTTP Live Streaming and RTSP are also supported. MPEG-DASH and Smooth Streaming are not yet supported.

World reconstruction

ML1 scans the space around it in real time. Data is collected from several positions, and then combined to reject noise. There are three types of data available to developers:

• getting tops;
• getting planes;
• getting the point of intersection of the ray with the object (raycast).

By default, recognized real-world objects will overlap augmented reality objects.

Recommendations for use:

• ML1 expects a static environment. If there is too much movement around you, there may be surprises;
• for a better crawl space walk around the room;
• do not move too fast;
• recommended distance to sensors from 40 cm to 5 meters;
• sunlight and other infrared light may adversely affect camera performance;
• sensors may not recognize black, shiny, and light-absorbing objects;

Simulator

The SDK also includes the Virtual Room Generator and the Magic Leap Remote. Magic Leap Remote - ML1 device simulator. Virtual Room Generator - a generator of rooms that can be transferred to the simulator.

The simulator allows you to work with audio, move your device in a virtual room, work with a virtual trackpad, recognize gestures, track images and get data about room objects using a raycast.

The company promises to deliver its devices to consumers later this year. You can still have time to prepare your project and be in the list of the first in the app store by the time sales start.
We will wait for news from Magic Leap and the responses of competitors. At the moment, only HoloLens from Microsoft is among them. But a few days ago Apple opened the registration for the WWDC conference, whose logo this time is very similar to 3D interfaces: