New Intel RealSense SDK 2016 R2 Features
The second release of the Intel RealSense SDK (for Windows) in 2016 (2016R2) is version 10.0.26.0396. The new version significantly improved the work of the background segmentation (BGS), hand tracking in pointer mode and 3D Scan module for the SR300 camera. The first two modules are already available in the form of a “golden” version, the 3DScan module for the SR300 has moved to the beta level. For the R200, some components of the Enhanced Photography (EP) module have moved to the “gold” level, including the EP Meaurement. At the same time, the modules EP Realtime Depth Enhancement and EP Tracking are still at the level of preliminary familiarization. The remaining functions of the R200 have not changed.
The minimum DCM versions for Intel RealSense cameras are increased: 2.1.24.9476 for the R200 camera, 3.1.25.1077 for the SR300 camera. Camera F200 requires version 1.4.27.41944 or later.
The supported version of Unity has been increased to 5.2.3.p3 or later (version 4.1 was previously supported). Note. Unity 5 requires 64-bit libraries.
Green Screen / BGS : now insertion (extinction) and additional callbacks are available for near and far limits and for user detection. An additional module of extended dynamic range has been added, other modifications of the algorithms have been implemented, which increase the accuracy of creating models of fingers and dark hair masks, the accuracy of detecting other artifacts. Problems with the recognition and determination of the distance to objects that absorb infrared radiation, transparent, very thin and black objects, as well as very light hair and hair whose color is close to the background color, are still preserved. Only one person can be recognized at a time; The quality of the mask is significantly reduced if several people appear within the range of the camera. For recorded video clips with an extended dynamic range, it may be necessary to delete the first frame in order to avoid stalling during real-time playback.
')
Cursor mode . Improved accuracy of gesture recognition, including clicking, rotation (in both directions), opening and squeezing the palm. You can reset the Adaptive Point to a specific location (ResetAdaptiveByID). The range is increased to 115 cm, and the speed is up to 2 m / s, with an error fixed at a detection range of 70 cm. Note. The default power supply is changed to OFF (you can set the value to On to save power by reducing the frame rate to a time when your hands are not in the field of view of the camera, but there is a delay of 2 seconds when your hands return to the field of view of the camera).
3DScan . maxTextureResolution can limit the size of the created texture; in flopPreviewImage, mirroring (horizontally) of the preview is implemented for the SR300 camera; useMarker has improved tracking of objects that are close in shape (cylinder, ball). In maxVertices, you can limit the quality of the vertices in the output three-dimensional model; Fixed rounding of depth values ​​in QueryVertices (front view cameras). In addition, in SetConfiguration for head detection using the SR300, the return message is changed from PXC_STATUS_NO_ERROR to PXC_STATUS_FEATURE_UNSUPPORTED. To scan a face without fiducial points using front-facing cameras, Feature_Unsupported returns; to avoid this, you need to load the Face Tracking module to obtain fiducial points or use the Variable scanning mode with SetArea ().
GestureConfiguration (PersonData) and ExpressionsConfiguration (PersonExpressions) have been added to PersonTracking Configuration / Data, as well as recognition of the location and position of the head, orientation of the person, and the bounding box of the head. At the same time, the configuration properties field was abolished (functions should be used to configure). Tracking can now use re-identification, and in the skeleton tracking mode, 6 points are tracked (and not 4).
LANDMARK_GROUP_RIGHT (| LEFT) _EAR has been added to FaceData, RecogntionData has been expanded to transmit data on the closest matches, QueryNumberOfMatches and QueryMatches have been changed.
The default SmoothingLevelType for face tracking has been changed from LANDMARKS_SMOOTHING_LOW to LANDMARKS_SMOOTHING_MEDIUM. Due to this, temporal smoothing is improved in frames.
PIXEL_FORMAT_DEPTH_CONFIDENCE has been added to PXC [M] Image , STREAM_OPTION_DEPTH_CONFIDENCE has been added to PXC [M] Capture.
PXC [M] Projection removed the incorrect rounding of the depth value in the QueryVertices API for front-view cameras; SelectOption can now return a projection map without cropping.
In the PXC [M] VideoModule interface, ProcessSample, SetProjection and SetGPUExec are extended (to determine CPU or GP usage).
EnhancedPhoto (EP) has added an insert to the surface (as well as to the plane), support for multiple stickers, AddSticker (), UpdateSticker () and RemoveSticker () along with identifier numbers for tracking stickers. You can also use PreviewEnhancedDepth () depth map preview in the video processing pipeline in each frame before use. For more information about problems in the EP and changes, see 2016R2 (release notes) , in the R200 known issues section (especially with regard to flicker).
In ScenePerception stream configurations, you can now use any sensor settings (gain, exposure, etc.) for both the color image and the depth map, including in Unity. IsGravitySensorSupportEnabled and IsInertialSensorSupportEnabled are also added to SP. You can turn off EnableSceneReconstruction for the first frame; EnableSceneReconstruction no longer turns on forcibly and automatically, so you can skip the embedding of the first frame in the three-dimensional model and volume. DoReconstruction supports integration of streams into a three-dimensional model, volume and voxels even before the transfer of the first frame after creating a module or after calling a reset. The thread safety of ExportSurfaceVoxels is implemented from the point of view of tracking: you can run ExportSurfaceVoxels in parallel with tracking.
Note. The .chm help file may not open. Use html or pdf files.
Unity (5.2.3.p3 or later with .NET 3.5) supports NativeTexturePlugin to update Unity Texture2D using Direct3D * 9 ​​or 11 or OpenGL * 2.0. Unity XML is located in the Plugins.Managed folder. In the SP Unity sample, you can switch to inline rendering, which provides a performance gain of about 50% for high-definition color images and avoids overloading streams when receiving tracking results. Note. If the VS libpxcclr.unity project builds incorrectly, click Project> Add Link and add [Unity installation folder] /Editor/Data/Managed/UnityEngine.dll.
The minimum DCM versions for Intel RealSense cameras are increased: 2.1.24.9476 for the R200 camera, 3.1.25.1077 for the SR300 camera. Camera F200 requires version 1.4.27.41944 or later.
The supported version of Unity has been increased to 5.2.3.p3 or later (version 4.1 was previously supported). Note. Unity 5 requires 64-bit libraries.
New opportunities
Green Screen / BGS : now insertion (extinction) and additional callbacks are available for near and far limits and for user detection. An additional module of extended dynamic range has been added, other modifications of the algorithms have been implemented, which increase the accuracy of creating models of fingers and dark hair masks, the accuracy of detecting other artifacts. Problems with the recognition and determination of the distance to objects that absorb infrared radiation, transparent, very thin and black objects, as well as very light hair and hair whose color is close to the background color, are still preserved. Only one person can be recognized at a time; The quality of the mask is significantly reduced if several people appear within the range of the camera. For recorded video clips with an extended dynamic range, it may be necessary to delete the first frame in order to avoid stalling during real-time playback.
')
Cursor mode . Improved accuracy of gesture recognition, including clicking, rotation (in both directions), opening and squeezing the palm. You can reset the Adaptive Point to a specific location (ResetAdaptiveByID). The range is increased to 115 cm, and the speed is up to 2 m / s, with an error fixed at a detection range of 70 cm. Note. The default power supply is changed to OFF (you can set the value to On to save power by reducing the frame rate to a time when your hands are not in the field of view of the camera, but there is a delay of 2 seconds when your hands return to the field of view of the camera).
3DScan . maxTextureResolution can limit the size of the created texture; in flopPreviewImage, mirroring (horizontally) of the preview is implemented for the SR300 camera; useMarker has improved tracking of objects that are close in shape (cylinder, ball). In maxVertices, you can limit the quality of the vertices in the output three-dimensional model; Fixed rounding of depth values ​​in QueryVertices (front view cameras). In addition, in SetConfiguration for head detection using the SR300, the return message is changed from PXC_STATUS_NO_ERROR to PXC_STATUS_FEATURE_UNSUPPORTED. To scan a face without fiducial points using front-facing cameras, Feature_Unsupported returns; to avoid this, you need to load the Face Tracking module to obtain fiducial points or use the Variable scanning mode with SetArea ().
GestureConfiguration (PersonData) and ExpressionsConfiguration (PersonExpressions) have been added to PersonTracking Configuration / Data, as well as recognition of the location and position of the head, orientation of the person, and the bounding box of the head. At the same time, the configuration properties field was abolished (functions should be used to configure). Tracking can now use re-identification, and in the skeleton tracking mode, 6 points are tracked (and not 4).
LANDMARK_GROUP_RIGHT (| LEFT) _EAR has been added to FaceData, RecogntionData has been expanded to transmit data on the closest matches, QueryNumberOfMatches and QueryMatches have been changed.
The default SmoothingLevelType for face tracking has been changed from LANDMARKS_SMOOTHING_LOW to LANDMARKS_SMOOTHING_MEDIUM. Due to this, temporal smoothing is improved in frames.
PIXEL_FORMAT_DEPTH_CONFIDENCE has been added to PXC [M] Image , STREAM_OPTION_DEPTH_CONFIDENCE has been added to PXC [M] Capture.
PXC [M] Projection removed the incorrect rounding of the depth value in the QueryVertices API for front-view cameras; SelectOption can now return a projection map without cropping.
In the PXC [M] VideoModule interface, ProcessSample, SetProjection and SetGPUExec are extended (to determine CPU or GP usage).
EnhancedPhoto (EP) has added an insert to the surface (as well as to the plane), support for multiple stickers, AddSticker (), UpdateSticker () and RemoveSticker () along with identifier numbers for tracking stickers. You can also use PreviewEnhancedDepth () depth map preview in the video processing pipeline in each frame before use. For more information about problems in the EP and changes, see 2016R2 (release notes) , in the R200 known issues section (especially with regard to flicker).
In ScenePerception stream configurations, you can now use any sensor settings (gain, exposure, etc.) for both the color image and the depth map, including in Unity. IsGravitySensorSupportEnabled and IsInertialSensorSupportEnabled are also added to SP. You can turn off EnableSceneReconstruction for the first frame; EnableSceneReconstruction no longer turns on forcibly and automatically, so you can skip the embedding of the first frame in the three-dimensional model and volume. DoReconstruction supports integration of streams into a three-dimensional model, volume and voxels even before the transfer of the first frame after creating a module or after calling a reset. The thread safety of ExportSurfaceVoxels is implemented from the point of view of tracking: you can run ExportSurfaceVoxels in parallel with tracking.
Note. The .chm help file may not open. Use html or pdf files.
Unity (5.2.3.p3 or later with .NET 3.5) supports NativeTexturePlugin to update Unity Texture2D using Direct3D * 9 ​​or 11 or OpenGL * 2.0. Unity XML is located in the Plugins.Managed folder. In the SP Unity sample, you can switch to inline rendering, which provides a performance gain of about 50% for high-definition color images and avoids overloading streams when receiving tracking results. Note. If the VS libpxcclr.unity project builds incorrectly, click Project> Add Link and add [Unity installation folder] /Editor/Data/Managed/UnityEngine.dll.
Reference materials
Source: https://habr.com/ru/post/312856/
All Articles