This article will help developers familiarize themselves with the capabilities of the RealSense plug- in for Unreal Engine 4, understand its architecture and device. This plug-in was developed by Intel to support the Intel® RealSense ™ SDK. This article is intended for experienced developers using UE4, as well as for developers who are familiar with RealSense technology. If you want to start exploring UE4, read the documentation and training materials . You can also download Epic Games Launcher here and view sample projects on the Learn tab.

Basic information and area of ​​this work

Unreal Engine 4 is the fourth generation of the Epic Games game engine. It was first introduced to a limited audience in 2012 at the Game Developers Conference. In 2014, all interested developers could already get the source code of this engine, paying a small monthly subscription. The subscription fee was canceled in 2015, now the source code of the entire engine is available to all developers for free.

The Unreal Engine source code is written in C ++. In the UE4 version, Epic developers first used Blueprint visual scripts instead of the Kismet system used in UE3. In fig. Figure 1 shows an example Blueprint script.


Figure 1. Sample Blueprint script in Unreal Engine * 4
')
Developers can create Blueprint nodes for events, functions, variables, macros, etc., and link them together using logical process chains. Blueprint scripts simplify the work of novice developers with limited programming experience and provide extremely fast prototyping.

Many UE4 games are implemented using a combination of C ++ program code and Blueprint scripts. Visual scripts allow you to manage animated images, define complex artificial intelligence behavior trees, and implement any kind of game logic.

Intel RealSense SDK Plugin

The Intel RealSense SDK was originally written in C ++, which gives developers a convenient way to access SDK functions from game code, also written in C ++. The described plug-in extends the development capabilities of UE4 by providing access to the SDK functionality through Blueprint scripts.
At the time of this writing, the plug-in officially supported the following functions:

• Transfer the raw color image and depth image from the camera.
• Three-dimensional scanning of faces and objects.

Work is also underway on implementing support for two more components:

• Head tracking.
• The perception of the scene.

The ultimate goal of the plug-in is to support all the features and capabilities of the full version of the Intel RealSense SDK applicable to games. For a complete list of RealSense SDK features, organized by camera and availability, see the RealSense SDK release notes .

The source code of the plug-in is available here .
In the sample project, there is a map designed to demonstrate each of the capabilities of the plug-in; commentary scripts describe the intended use of RealSense components.
If you are interested in developing using any of the experimental functions of the plug-in, use additional branches in the plug-in storage.

In addition, the Getting Started guide is located in the project folder to help add a plug-in to a new project. The same actions are described in the next training video.

RealSense components

This section provides a more detailed description of each of the RealSense components.

UE4 components

The Unreal Engine has a special type of object called a component. Its purpose is to join other objects. Components are usually quite compact and autonomous, they provide a limited set of functions to the objects to which they are attached. They can be attached to almost any imaginable objects, including player characters, static objects, vehicles, lights, and interface elements. The purpose of the components is to provide similar functionality to dissimilar classes of objects.

To get an idea of ​​the latitude of the components, for example, consider the Scene component that applies the transform to the object to which it is attached. Transformation is the application of location, rotation and scale to an object. Using the transformation, the developer can place the object in any place of the game world. This component makes it very easy to add a transformation to anything, even to light and sounds.

The RealSense plug-in is organized as a set of components, each of which provides access to specific features of the RealSense SDK. For example, the Camera Streams component provides a developer with access to a color image stream and a depth stream transmitted from a camera. In the UE4 editor, the list of components is in the Add Component drop-down menu in the Components window in the RealSense category, as shown in Figure. 2


Figure 2. Components of the RealSense plug-in in the Components window of the Unreal Engine * 4

RealSense component

This component is the abstract parent class of all other RealSense components. It provides the overall functionality needed by each component, but it’s impossible to attach it to any object in Unreal.
The data provided by this class include the following (Figure 3).

float ColorHorizontalFOV; //       float ColorVerticalFOV; //       float DepthHorizontalFOV; //       float DepthVerticalFOV; //       ECameraModel CameraModel; //   : F200, R200  SR300 FString CameraFirmware; //     

Figure 3. Member variables of the RealSense component

The RealSense component also provides the following features (Figure 4).

 void StartCamera(); //   RealSense void StopCamera(); //   RealSense bool IsCameraRunning(); //   true,    //       FStreamResolution GetColorCameraResolution(); virtual void SetColorCameraResolution(EColorResolution); //       FStreamResolution GetDepthCameraResolution(); virtual void SetDepthCameraResolution(EDepthResolution); //   true,       . bool IsStreamSetValid(EColorResolution, EDepthResolution); 

Figure 4. Member functions of the RealSense component

Component Camera Streams

This component provides access to the raw color image and depth data from an Intel RealSense camera, as well as convenient Texture objects to display these images.
The data provided by this class include the following (Figure 5).

 TArray<FSimpleColor> ColorBuffer; //  32-   (RGBA) TArray<int32> DepthBuffer; //  16-   ( ) UTexture2D* ColorTexture; //     . UTexture2D* DepthTexture; //     . 

Figure 5. Camera Streams member variable

The Camera Streams component also provides the following features that are overridden from the RealSense component:

 //    RealSense,  ColorBuffer  ColorTexture   . virtual void SetColorCameraResolution(EColorResolution) override; //    RealSense,  DepthBuffer  DepthTexture   . virtual void SetDepthCameraResolution(EDepthResolution) override; 

There is a small library of utility functions, it is called RealSense Utilities. One of these functions accepts a color buffer (Color Buffer object) and color texture (Color Texture object) as input data and fills this texture with buffer data. There is also a comparison function that works with a depth buffer (Depth Buffer object) and a depth texture (Depth Texture object). The use of this function is shown in Fig. 6


Figure 6. An example of using the Color Buffer to Texture function

Check out this quick tutorial to learn how to use the Camera Streams component of the RealSense plug-in to access a stream of color from your camera and display this image on the screen.

Scan 3D component

This component provides access to the intermediate 3D scanning module in the Intel RealSense SDK. Currently, the plug-in supports scanning of faces and objects, but does not support the output of textures, only vertex colors are displayed.
The data provided by this class include the following (Figure 7).

 TArray<FSimpleColor> ScanBuffer; //  32-   (RGBA),    ,      . UTexture2D* ScanTexture; //    ScanBuffer TArray<FVector> Vertices; //   Vector3,     TArray<int32> Triangles; //   ,   TArray<FColor> Colors; //  ,    

Figure 7. The variable members of the Scan 3D component

The Scan 3D component also provides the following functions (Figure 8).

 //    (    )  , ,       void ConfigureScanning(EScan3DMode, bool); void StartScanning(); //    void StopScanning(); //    void ResetScanning(); //    bool IsScanning();//   true,     void SaveScan(FString); //     OBJ-    . void LoadScan(FString); //      OBJ- 

Figure 8. Member functions of the Scan 3D component

Finally, the Scan 3D component provides one Blueprint event (Figure 9).

 OnScanComplete(); // ,      

Figure 9. Blueprint OnScanComplete Event Node

The SaveScan () function instructs the scan engine to save and return immediately after that. Somewhat later (in the new camera frame), the RealSense plug-in stream will perform the save operation. Developers should use the OnScanComplete () event to receive notification that the save is complete when scanning.

Architecture

RealSense Session Manager

When a component is attached to an object in UE4, this object receives a new instance of this component class. You can create multiple instances of the same component class and attach them to multiple objects within a single project. But for the RealSense components, this causes a problem. The fact is that there is only one Intel® RealSense ™ camera in the system that needs to be managed by the developer. But since you can create multiple instances of the same component class, each instance of a RealSense component will receive its own copy of the data from the camera.

To solve this problem, the RealSense Session Manager object is used in the RealSense plug-in. This is a singleton class, an instance of which is automatically created when the first RealSense component is added to an object in the game. All subsequent instances of the RealSense component receive a link to the RealSense Session Manager when they are initialized. RealSense Session Manager owns the master copy of all Intel RealSense SDK data. Each RealSense component reads data from RealSense Session Manager and writes data to it.

Multithreading

RealSense Session Manager is responsible for managing the dedicated stream of the RealSense plug-in. This stream provides RealSense SDK modules so that the game uses RealSense component classes based on the SDK capabilities. This stream is where the AcquireFrame () / ReleaseFrame () SDK works. All involved modules process the received frames.

RealSense Session Manager uses a triple buffer to control the exchange of data between the RealSense plug-in stream and the main stream of the UE4 game. The RealSense plug-in stream writes all new data to the background buffer. After calling ReleaseFrame (), the background buffer exchanges data with the middle buffer, locked by means of mutually exclusive locks (mutexes). After that, in each clock of the game stream, the foreground frame is exchanged with the middle buffer, providing the latest data to the Session Manager and to all RealSense components.

Conclusion

The RealSense plug-in provides developers using UE4 a simple and easy to use interface to the Intel RealSense SDK. The features of the SDK are made available to the Blueprint scripting system, which significantly speeds development. Thanks to the architecture based on a set of components, developers can add 3D capabilities to almost any object in the game.