Intel RealSense SDK Architecture

Intel RealSense SDK differs in its architecture from its predecessor - Intel Perceptual Computing SDK. If you, as a developer, used the Intel Perceptual Computing SDK to develop applications, you will quickly appreciate the improved programming model in the new SDK, which provides access to supported modes of operation through several popular application development platforms. In this article, we describe the major changes in the Intel RealSense SDK.

Architecture Overview

The SDK core, I / O module, and algorithm modules form the core of the SDK. The SDK core helps control the application execution pipeline and I / O modules (cameras). Algorithm modules form an intermediate level for hand tracking, gesture recognition, face tracking, speech recognition and other modes of operation. These algorithms provide interfaces for developing applications using development platforms — C ++, C #, Unity *, Java *, Processing *, and so on. At the top of this set of technologies are the Intel RealSense SDK applications (Figure 1).


Figure 1. Intel RealSense SDK Architecture

This improvement will be very obvious to application developers familiar with the Intel Perceptual Computing SDK. Previous versions of the SDK interfaces could only be used on Unity and Java platforms (Fig. 2).
')

Figure 2. Intel Perceptual Computing SDK Architecture

Most of the functions of the SDK were available only to C ++ / C # developers, which made Unity and other application development platforms under adverse conditions. The Intel RealSense SDK removes this restriction by providing uniform access to the core and middleware capabilities through interfaces designed for each platform. In fig. 3 it can be seen that the support of DLL libraries for all platforms has been completely redone. The C ++ / C # interface is now available through the PInvoke interface instead of the C ++ / CLI C # shell in the Intel Perceptual Computing SDK. If we talk about Unity, the interface based on PXCUPipeline in the old SDK is replaced by the C # interface based on PInvoke. For Java and Processing platforms, the PXCUPipeline based interface has been replaced with a completely redesigned JNI shell.


Figure 3. Uniform API access across all application development platforms

Simplified class hierarchy

Figure 4. Intel RealSense SDK Class Hierarchy

The interface structure in the Intel Perceptual Computing SDK was highly hierarchical. To perform simple tasks, a long sequence of initialization, configuration, and data acquisition operations was required. In Intel RealSense SDK, this hierarchical structure is simplified, providing more convenient access to various functions and modes of operation.
As shown above in fig. 4, PXC [M] SenseManager replaces the UtilPipeline class of the SDK core. SenseManager is responsible for organizing and managing the execution pipeline. PXC [M] CaptureManager manages all camera and stream devices instead of the UtilCapture interface in the Perceptual Computing SDK. Note that the PXC [M] Capture interface also provides more convenient access to depth data. The Intel Perceptual Computing SDK needed to access two different streams to obtain depth and vertex data. In the Intel RealSense SDK, this data is available in one stream. The PXC [M] Face3D interface controls the face tracking module instead of the PXC [M] Face interface, and the PXC [M] HandAnalysis handles gestures and hands tracking instead of the PXC [M] Gesture interface.
For more information about these modules, see the next section, the transition guide for new APIs.

New API Transition Guide

Not only the whole set of SDKs has been redesigned, but over 50% of the APIs from the Intel Perceptual Computing SDK have been redesigned into the Intel RealSense SDK. Most of the changes are due to the emergence of support for three-dimensional scenes in the Intel RealSense SDK: due to this, some existing modes, such as face and hand tracking, have been improved. Consider some of these improvements.

Face tracking module

In terms of capabilities, the face tracking module now supports 78 reference points with different facial expressions in two-dimensional and three-dimensional modes. For comparison: in the Intel Perceptual Computing SDK no more than 7 control points were supported. The introduction of the depth parameter increases the reliability of the data obtained. In addition, in the Intel Perceptual Computing SDK, it was required to individually configure face detection, frame detection, and face recognition, which made the entire face analysis process extremely cumbersome. The face analysis module and its associated APIs have been redesigned to reflect these aspects. Instead of the PXC [M] FaceAnalysis module, the PXC [M] Face3D module with a single-level structure is used. It is enough to configure this module only once before receiving face detection values, fixed points, facial expressions and face recognition values ​​(Fig. 5).


Figure 5. Comparison of the face analysis module in the Intel Perceptual Computing SDK and in the Intel RealSense SDK

Hand tracking module

The hand tracking module has been significantly enhanced in the Intel RealSense SDK. Compared to data for 7 hand points and 10 standard gestures, Intel RealSense SDK now supports data for 22 points, finger identification, right and left hand identification with orientation and rotation parameters for three-dimensional interaction, as well as a set of standard gestures. The PXC [M] Gesture interface used in the Intel Perceptual Computing SDK has been replaced by the PXC [M] HandAnalysis interface for more convenient access to hand data.



Table 1 (see below) summarizes the major improvements in the Intel RealSense SDK. We encourage developers to read the SDK reference manual, which is part of the beta, for more detailed information about each API in these interfaces.

Table 1. Comparison of the Intel Perceptual Computing SDK and the Intel RealSense SDK

Categories	Intel Perceptual Computing SDK	Intel RealSense SDK	Note
Camera	Creative device properties only		Not supported in IVCAM
	Vertex format	Depth / DepthRaw / DepthFloat / VerticesFloat	PXCImage provides the ability to convert on the fly.
	NOJR.MAP NO.UV.MAP	REQUIREJR.MAP REQUIRE_UV_MAP	Change default
	PXCProjection	PXCProjection	Half of the functions redone.
MW components	PXCGesture	PXCHandAnalysis	Brand new interface
	PXCFace	PXCFace3D	Brand new interface
	Utilcapture	PXCCaptureManager	New interface
	Utilpipeline	PXCSenseManager	New interface
Language and Platform Support	Pxcupipeline-based Unity / Java / Proce ssing platform support	P / Invoke C # Shell	Completely redesigned.
	C ++ shell / C # command line interface	P / Invoke C # Shell	Completely redesigned.
	PXCUPipeline based Java / Processing support	New JNI shell	Completely redesigned.
Documentation	Reworked.

Conclusion

The Intel RealSense SDK has a number of advantages over the previous generation Intel Perceptual Computing SDK. Improved many existing modes of operation, such as face and hand tracking algorithms; Improved API access mechanism for most supported application development platforms. Thanks to the uniformity of access to the API and the improvements at the intermediate level, a very convenient platform has emerged through which application developers can master new possibilities of interaction with users.

Additional materials

Intel RealSense Technology Review .
User interaction design guides .