3D scanning using Intel RealSense

The newly released Intel RealSense F200 Camera SDK includes 3D scanning. This is an amazing feature that will allow developers and computer graphics specialists to scan real objects and apply them in their projects. One example of using this technology is scanning real objects for use in the Unity game engine. In this article I will introduce you to the details of this process.

What do you need?

• Intel RealSense camera with a fourth-generation Intel Core processor or higher with an SDK installed (SDK is distributed free of charge)
• Object scanned using Intel RealSense technology and converted to a PLY format file (using the free MeshLab program)
• Blender (freeware)

Scan using the Intel RealSense SDK

Install the Intel RealSense SDK. Open the express button panel in Windows and find RealSense. In the list you will see the RealSense SDK Sample Browser. Right-click it and select Run as Administrator.
')


In the Sample Browser, you will see the Common Samples tab. These are examples that work with both the F200 camera and the R200 camera. Find an example called 3D Scan (C ++).



What you need to know when 3D-scanning

• This example is designed to trace a rigid three-dimensional object in space as the object rotates in front of the camera, which allows you to scan the object at different angles.
• If one part of the object rotates and the other does not rotate, it will make tracing difficult and will result in a poor quality scan. For example, if you scan your head, do not turn it relative to the neck, otherwise it will worsen the quality of the scan. Instead, it is better to turn the entire torso so that the camera perceives it as a single object.
• You can lean forward and backward, turn the torso left and right and make a full turn. Over time, you can learn how to scan objects by turning them 360 degrees.
• Scan objects with uniform illumination. Avoid shadows, since shadows or reflections of light will also be included in the 3D texture.

Stages scan.

In the Sample Browser, select Run.

• Place the object in front of the camera and you will see its display in the scanner. Wait until the second window appears.
• Once the second window appears, start rotating the object in one direction, then in the other, and then 360 degrees. After that, tilt the object backward, then forward, then hold it in the starting position. If you are scanning a face, for the last few seconds you should look straight ahead over the camera so that your eyes are fixed in one position.
• When the scan is complete, a web browser window opens where you can move and rotate the resulting image. The image is saved in your Documents folder under the name 3Dscan.obj.
• You can use this example an unlimited number of times to create high-quality scans that you can refine and use in your applications or games.

Convert files to PLY format

When scanning files are saved by default in the format of OBJ. You can convert a file to PLY format with the free MeshLab tool.

Download and install MeshLab (free).

From the File menu, select Import. Select the 3Dscan.obj file located in the root of the Documents folder.



The imported object can be rotated to the side. You can left-click on the bottom of the object and drag to rotate the object and examine it. You will see a grid with a colored texture. Color information is stored in Vertex Colors format, i.e., each vertex is colored with pixels in RGB colors obtained by the camera during scanning.



From the File menu, select Export Mesh As.
In the dialog box, select Stanford Polygon File Format (* .ply).



When saving, specify the path that you can then find.
Now you can go to work in Blender, where you can clear the grid and convert the colors of Vertex Colors to a UV map format.

Vertex colors and UV map in Blender

The point is to import a three-dimensional model file and create a copy of it. Then you need to clear the copy and use the colors of the vertices as the UV map in the new version of the model.

Create a new project in Blender, select Import PLY and select the PLY file exported from MeshLab.



After importing, you will see that the textures are missing.



If you switch to the Texture view, you will see that the RGB colors from the scanned model are now associated with the Texture mode.



To display colors correctly, go to the Materials view. In the right pane, select the Materials icon. Create a new material. In the panel editor, set the Shading parameter to Shadeless, and in the Options section, select the Vertex Color Paint check box. The colors of the vertices will be assigned to the material of the object.



As you can see, it is impossible to work with such a model. It is full of holes, and the number of polygons far exceeds what is needed. You need to create a version of this model that can be cleaned, and present the texture with the colors of the vertices in the form of a UV map. Here we also import this version into Unity.



Create a duplicate model

Go to the Object menu. Select Duplicate Mesh and press Enter. Do not move the mouse.



Go to the Object menu, select Move To Layer and select the box on the right.



Go to Resource Explorer and rename the models. For example, let the first model be called the “original model”, and the second “the second model”. This will help keep order.
Select the second model, select the level to view, and switch the mode to Solid rendering.
Then click the Tool Wrench icon to add a modifier. We will change the number of polygons in the second model and, thus, make it cleaner.
Select Remesh in the Tools menu. In the Remesh toolbox, specify a value from 7 to 8 for the Oct Tree parameter. Select Smooth mode, select the Smooth Shading checkbox. At the same time comparison with materials will disappear.



Click Apply. Now we have a cleaner model to which we can apply Vertex and UVMap.



Switch to the edit mode of this second model. From the Select menu, choose Select All.



Create a new window and set it to UV Map to display the UVMap.



In the 3D window, go to the Mesh menu and select UV.



In the UV Map window, select NEW to create a texture. Uncheck the Alpha box. Specify the name of the model to be exported for the texture.
Switch to Object mode. You will see a black model and a black UV map. This is normal.
Now turn on both layers with the Shift key.



In the Resource Explorer, select the original model.
Click the camera icon.
In the Bake area, select the Select to Active check box. Make sure the Bake to Vertex checkbox is unchecked.



In the Scene section, set the Color Management parameter to None.



In the Asset menu, first select the original model. Then hold down the SHIFT key and select the second model. (In that order.) Both layers should be visible when the eye and camera icons are on.
Click the camera icon. In the Bake section, click Bake.



Now the texture should be mapped to the map. To verify this, turn off the first layer, so that only the layer with the second model remains. Go to the Texture view and you will see a model on which the UV map is superimposed as a texture.



Pay attention to some instability of the image in the missing places of the model.



You can switch to Texture Paint mode and correct these shortcomings. You can also now switch to Sculpt mode and fill in the missing places, clean the model or add new elements to it. Now the model is ready, it can be exported. And the last thing to remember.

• Make sure the UVMap card has the same name and is in the same folder as the model.
• Go to the UVMap window, on the Image menu, select Save As and select the folder where you want to save the model.
• Then go to the File menu and export the file to the .FBX format, specifying the same name as for the UV map.

Now everything is ready to import this three-dimensional model into Unity.

Import Files to Unity

After completing the steps in the previous section, you should have got a model in the .FBX format with a UV texture map.

Open Unity (Unity 5 is available for free download).
Create a new project and scene.



In the Assets menu, select Import new asset. Select the model in the .FBX format that you exported from Blender.

When importing an object, it is quite possible that its texture is not imported.
In this case, you will see a light gray version of the grid. This is normal. Now we will add a texture map.



Select the Materials folder, right-click and select Import Asset. Select the PNG file you saved in Blender and import it.

You will now see a texture map next to the gray material. Right-click on the gray material in the form of a ball, whose name matches the name of your mesh, and remove the material.




Return to the Assets folder and right-click on the grid you imported. Select Reimport.



After re-import, the material is created correctly and the grid is displayed with the desired texture.



If the model is displayed in pink in the viewport, simply select another object, such as a camera, and then select your grid again. Now it should be displayed correctly.



The object is now ready for use. Drag the object onto the stage. You can zoom in and adjust the camera to get the look you want.

By moving the camera, you should see your object correctly displayed in Unity.



Finally, if you want to edit a texture, for example, to make it more or less shiny, go to the Materials folder and select the spherical version of your texture. In the Inspector window, you can edit the degree of reflection and smoothness to get the look of the texture you want.

Good luck and waiting for your comments! You can also find me on Twitter: @bobduffy.