Virtual quadrocopter on Unity + OpenCV (Part 2)

Good day, dear habravchane!

With this article, I would like to continue the series on how to make friends with Unity, C ++ and OpenCV. As well as how to get a virtual environment for testing computer vision algorithms and navigating drones based on Unity. In the previous article, I talked about how to make a virtual quadrocopter in Unity. This article will discuss how to connect the C ++ plugin, transfer the image from the virtual camera there and process it using OpenCV.

Idea

The idea is to render the camera image into texture. Further, this texture is loaded into the memory of the video card in Unity, from where it is extracted via OpenGL already inside C ++ and loaded into the OpenCV matrix. Then you can analyze / process it with OpenCV. And in the end, you can transfer the modified texture back to Unity by loading it in place of the corresponding texture into the memory of the video card. So let's get started.
')

Render camera in texture

First we need to create textures in which images from cameras will be placed. To do this, click the right button in the assets folder, then Create -> Render Texture. Next we need to add a camera to our quadric. This is done in the Hierarchy, using Create -> Camera. Add our camera to Quadrocopter -> Frame so that it attaches to the frame. In the Target Texture camera property, we specify the Render Texture that we created. We would also like to see the camera texture in Unity. To do this we need Create -> UI -> Canvas in our scene. This object is used to display controls (usually two-dimensional) on the screen. I called it CamerasTextures. In the canvas, we need to add a UI -> RawImage. In the Texture parameter of the image we specify our texture. Using Rect Transform inside RawImage we position the image to the right place on the screen; this is easiest to do in the Game tab. We start and see how the picture from the camera changes while the quadrocopter is flying.
Virtual quadcopter with camera

Texture Transfer in C ++

Official plugins documentation can be found here . Official examples may also be helpful: this and this . I will transfer and receive from C ++ textures in a separate script. Let's call it GameManager. To make it work, you need to add an empty object to the scene and add this script to it. Below, I first describe the individual parts of the script, then I will give the whole script.

Function passing the texture ID in C ++

//  ,     . //          private Texture2D cam1Tex; //         private void PassCamerasTexturesToPlugin () { //   cam1Tex = new Texture2D(256,256,TextureFormat.ARGB32,false); //   cam1Tex.filterMode = FilterMode.Point; //  Apply()    GPU cam1Tex.Apply(); //       . // ,   ,     GameObject.Find ("/CamerasTextures/Camera1RawImage").GetComponent<RawImage>().texture = cam1Tex; // ,        #if UNITY_GLES_RENDERER SetTextureOfCam1 (cam1Tex.GetNativeTexturePtr(), cam1Tex.width, cam1Tex.height); #else SetTextureOfCam1 (cam1Tex.GetNativeTexturePtr()); #endif } //        //      #if UNITY_IPHONE && !UNITY_EDITOR [DllImport ("__Internal")] #else //      ,     [DllImport ("QuadrocopterBrain")] #endif #if UNITY_GLES_RENDERER private static extern void SetTextureOfCam1(System.IntPtr texture, int w, int h); #else private static extern void SetTextureOfCam1(System.IntPtr texture); #endif

A feature that will update the cam1Tex texture with camera data.

  private IEnumerator CallPluginAtEndOfFrames () { while (true) { //      yield return new WaitForEndOfFrame(); RenderTexture cam1RT = GameObject.Find ("/Quadrocopter/Frame/Camera1").GetComponent<Camera>().targetTexture; //  Render Texture -  ,      RenderTexture.active = cam1RT; cam1Tex.ReadPixels(new Rect(0, 0, cam1RT.width, cam1RT.height), 0, 0); //    GPU cam1Tex.Apply (); RenderTexture.active = null; //     //  int     //       . //        debug  GL.IssuePluginEvent(GetRenderEventFunc(), frameIndex++); } } //  ,    , //      C++ #if UNITY_IPHONE && !UNITY_EDITOR [DllImport ("__Internal")] #else [DllImport("QuadrocopterBrain")] #endif private static extern IntPtr GetRenderEventFunc(); }

All script code should look like this.

 using UnityEngine; using UnityEngine.UI; using System; using System.Collections; using System.Runtime.InteropServices; public class GameManager : MonoBehaviour { private Texture2D cam1Tex; private int frameIndex = 0; IEnumerator Start () { PassCamerasTexturesToPlugin (); yield return StartCoroutine ("CallPluginAtEndOfFrames"); } private IEnumerator CallPluginAtEndOfFrames () { while (true) { //      yield return new WaitForEndOfFrame(); RenderTexture cam1RT = GameObject.Find ("/Quadrocopter/Frame/Camera1").GetComponent<Camera>().targetTexture; //  Render Texture -  ,      RenderTexture.active = cam1RT; cam1Tex.ReadPixels(new Rect(0, 0, cam1RT.width, cam1RT.height), 0, 0); //    GPU cam1Tex.Apply (); RenderTexture.active = null; //     //  int     //       . //        debug  GL.IssuePluginEvent(GetRenderEventFunc(), frameIndex++); } } private void PassCamerasTexturesToPlugin () { //   cam1Tex = new Texture2D(256,256,TextureFormat.ARGB32,false); //   cam1Tex.filterMode = FilterMode.Point; //  Apply()    GPU cam1Tex.Apply(); //       . // ,   ,     GameObject.Find ("/CamerasTextures/Camera1RawImage").GetComponent<RawImage>().texture = cam1Tex; // ,        #if UNITY_GLES_RENDERER SetTextureOfCam1 (cam1Tex.GetNativeTexturePtr(), cam1Tex.width, cam1Tex.height); #else SetTextureOfCam1 (cam1Tex.GetNativeTexturePtr()); #endif } //        //      #if UNITY_IPHONE && !UNITY_EDITOR [DllImport ("__Internal")] #else //      ,     [DllImport ("QuadrocopterBrain")] #endif #if UNITY_GLES_RENDERER private static extern void SetTextureOfCam1(System.IntPtr texture, int w, int h); #else private static extern void SetTextureOfCam1(System.IntPtr texture); #endif //  ,    , //      C++ #if UNITY_IPHONE && !UNITY_EDITOR [DllImport ("__Internal")] #else [DllImport("QuadrocopterBrain")] #endif private static extern IntPtr GetRenderEventFunc(); }

Opencv

Since we are going to use OpenCV for image processing you will need to install it yourself. The usual vanilla OpenCV, taken from the official site, will do . I put it myself on the guide . If you are trying to repeat what is written in the article, I advise you to start and make and compile an empty dynamic library project that would use OpenCV in any way. The above guide should help you. You will also need OpenGL functions.

An example of an empty library

 #include <opencv2/opencv.hpp> void someFunc () { cv::Mat img (256, 256, CV_8UC4); return 0; }

Getting and processing textures in the plugin

Here, too, to begin with, I will cite separate fragments of the program, and then the entire code is already complete.

Texture ID Transfer Function

 //     static void* g_Cam1TexturePointer = NULL; #ifdef SUPPORT_OPENGLES static int g_TexWidth = 0; static int g_TexHeight = 0; #endif //    . //   C#  #ifdef SUPPORT_OPENGLES extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API SetTextureOfCam1(void* texturePtr, int w, int h) #else extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API SetTextureOfCam1(void* texturePtr) #endif { g_Cam1TexturePointer = texturePtr; #ifdef SUPPORT_OPENGLES g_TexWidth = w; g_TexHeight = h; #endif }

Texture processing function

 static void UNITY_INTERFACE_API OnRenderEvent(int eventID) { //     RenderingPluginExampleXX.zip, //        ,     OpenGL. //       ,      //    #if SUPPORT_OPENGL if (g_Cam1TexturePointer) { GLuint gltex = (GLuint)(size_t)(g_Cam1TexturePointer); glBindTexture (GL_TEXTURE_2D, gltex); int texWidth, texHeight; glGetTexLevelParameteriv (GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &texWidth); glGetTexLevelParameteriv (GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, &texHeight); //  OpenCV,      cv::Mat img (texHeight, texWidth, CV_8UC4); //     glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE, img.data); //         Cam1 cv::putText(img, "Cam1", cv::Point2f(10,50), cv::FONT_HERSHEY_SIMPLEX, 2, cv::Scalar(0, 0, 0, 255), 3); //     ,     Unity. // Unity     GUI  OpenCV,    imshow    //           Unity glTexSubImage2D (GL_TEXTURE_2D, 0, 0, 0, texWidth, texHeight, GL_RGBA, GL_UNSIGNED_BYTE, img.data); } #endif } //  GameManager.cs extern "C" UnityRenderingEvent UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API GetRenderEventFunc() { return OnRenderEvent; }

My plugin consists of 2 files:

Main.h

 #ifndef __QuadrocopterBrain__Main__ #define __QuadrocopterBrain__Main__ // Which platform we are on? #if _MSC_VER #define UNITY_WIN 1 #elif defined(__APPLE__) #if defined(__arm__) #define UNITY_IPHONE 1 #else #define UNITY_OSX 1 #endif #elif defined(__linux__) #define UNITY_LINUX 1 #elif defined(UNITY_METRO) || defined(UNITY_ANDROID) // these are defined externally #else #error "Unknown platform!" #endif // Which graphics device APIs we possibly support? #if UNITY_METRO #define SUPPORT_D3D11 1 #elif UNITY_WIN #define SUPPORT_D3D9 1 #define SUPPORT_D3D11 1 // comment this out if you don't have D3D11 header/library files #ifdef _MSC_VER #if _MSC_VER >= 1900 #define SUPPORT_D3D12 1 #endif #endif #define SUPPORT_OPENGL 1 #elif UNITY_IPHONE || UNITY_ANDROID #define SUPPORT_OPENGLES 1 #elif UNITY_OSX || UNITY_LINUX #define SUPPORT_OPENGL 1 #endif #endif /* defined(__QuadrocopterBrain__Main__) */

Main.cpp

 #include <math.h> #include <stdio.h> #include <vector> #include <string> #include "Unity/IUnityGraphics.h" #include <opencv2/opencv.hpp> #include "Main.h" // -------------------------------------------------------------------------- // Include headers for the graphics APIs we support #if SUPPORT_D3D9 #include <d3d9.h> #include "Unity/IUnityGraphicsD3D9.h" #endif #if SUPPORT_D3D11 #include <d3d11.h> #include "Unity/IUnityGraphicsD3D11.h" #endif #if SUPPORT_D3D12 #include <d3d12.h> #include "Unity/IUnityGraphicsD3D12.h" #endif #if SUPPORT_OPENGLES #if UNITY_IPHONE #include <OpenGLES/ES2/gl.h> #elif UNITY_ANDROID #include <GLES2/gl2.h> #endif #elif SUPPORT_OPENGL #if UNITY_WIN || UNITY_LINUX #include <GL/gl.h> #else #include <OpenGL/gl.h> #endif #endif // Prints a string static void DebugLog (const char* str) { #if UNITY_WIN OutputDebugStringA (str); #else fprintf(stderr, "%s", str); #endif } //     static void* g_Cam1TexturePointer = NULL; #ifdef SUPPORT_OPENGLES static int g_TexWidth = 0; static int g_TexHeight = 0; #endif //    . //   C#  #ifdef SUPPORT_OPENGLES extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API SetTextureOfCam1(void* texturePtr, int w, int h) #else extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API SetTextureOfCam1(void* texturePtr) #endif { g_Cam1TexturePointer = texturePtr; #ifdef SUPPORT_OPENGLES g_TexWidth = w; g_TexHeight = h; #endif } static void UNITY_INTERFACE_API OnRenderEvent(int eventID) { //     RenderingPluginExampleXX.zip, //        ,     OpenGL. //       ,      //    #if SUPPORT_OPENGL if (g_Cam1TexturePointer) { GLuint gltex = (GLuint)(size_t)(g_Cam1TexturePointer); glBindTexture (GL_TEXTURE_2D, gltex); int texWidth, texHeight; glGetTexLevelParameteriv (GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &texWidth); glGetTexLevelParameteriv (GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, &texHeight); //  OpenCV,      cv::Mat img (texHeight, texWidth, CV_8UC4); //     glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE, img.data); //         Cam1 cv::putText(img, "Cam1", cv::Point2f(10,50), cv::FONT_HERSHEY_SIMPLEX, 2, cv::Scalar(0, 0, 0, 255), 3); //     ,     Unity. // Unity     GUI  OpenCV,    imshow    //           Unity glTexSubImage2D (GL_TEXTURE_2D, 0, 0, 0, texWidth, texHeight, GL_RGBA, GL_UNSIGNED_BYTE, img.data); } #endif } // -------------------------------------------------------------------------- // UnitySetInterfaces static void UNITY_INTERFACE_API OnGraphicsDeviceEvent(UnityGfxDeviceEventType eventType); static IUnityInterfaces* s_UnityInterfaces = NULL; static IUnityGraphics* s_Graphics = NULL; static UnityGfxRenderer s_DeviceType = kUnityGfxRendererNull; extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API UnityPluginLoad(IUnityInterfaces* unityInterfaces) { DebugLog("--- UnityPluginLoad"); s_UnityInterfaces = unityInterfaces; s_Graphics = s_UnityInterfaces->Get<IUnityGraphics>(); s_Graphics->RegisterDeviceEventCallback(OnGraphicsDeviceEvent); // Run OnGraphicsDeviceEvent(initialize) manually on plugin load OnGraphicsDeviceEvent(kUnityGfxDeviceEventInitialize); } extern "C" void UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API UnityPluginUnload() { s_Graphics->UnregisterDeviceEventCallback(OnGraphicsDeviceEvent); } // -------------------------------------------------------------------------- // GraphicsDeviceEvent // Actual setup/teardown functions defined below #if SUPPORT_D3D9 static void DoEventGraphicsDeviceD3D9(UnityGfxDeviceEventType eventType); #endif #if SUPPORT_D3D11 static void DoEventGraphicsDeviceD3D11(UnityGfxDeviceEventType eventType); #endif #if SUPPORT_D3D12 static void DoEventGraphicsDeviceD3D12(UnityGfxDeviceEventType eventType); #endif #if SUPPORT_OPENGLES static void DoEventGraphicsDeviceGLES(UnityGfxDeviceEventType eventType); #endif static void UNITY_INTERFACE_API OnGraphicsDeviceEvent(UnityGfxDeviceEventType eventType) { UnityGfxRenderer currentDeviceType = s_DeviceType; switch (eventType) { case kUnityGfxDeviceEventInitialize: { DebugLog("OnGraphicsDeviceEvent(Initialize).\n"); s_DeviceType = s_Graphics->GetRenderer(); currentDeviceType = s_DeviceType; break; } case kUnityGfxDeviceEventShutdown: { DebugLog("OnGraphicsDeviceEvent(Shutdown).\n"); s_DeviceType = kUnityGfxRendererNull; g_Cam1TexturePointer = NULL; break; } case kUnityGfxDeviceEventBeforeReset: { DebugLog("OnGraphicsDeviceEvent(BeforeReset).\n"); break; } case kUnityGfxDeviceEventAfterReset: { DebugLog("OnGraphicsDeviceEvent(AfterReset).\n"); break; } }; #if SUPPORT_D3D9 if (currentDeviceType == kUnityGfxRendererD3D9) DoEventGraphicsDeviceD3D9(eventType); #endif #if SUPPORT_D3D11 if (currentDeviceType == kUnityGfxRendererD3D11) DoEventGraphicsDeviceD3D11(eventType); #endif #if SUPPORT_D3D12 if (currentDeviceType == kUnityGfxRendererD3D12) DoEventGraphicsDeviceD3D12(eventType); #endif #if SUPPORT_OPENGLES if (currentDeviceType == kUnityGfxRendererOpenGLES20 || currentDeviceType == kUnityGfxRendererOpenGLES30) DoEventGraphicsDeviceGLES(eventType); #endif } // -------------------------------------------------------------------------- // GetRenderEventFunc, an example function we export which is used to get a rendering event callback function. extern "C" UnityRenderingEvent UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API GetRenderEventFunc() { return OnRenderEvent; }

How to put it all together

Create the folder in the Assets folder Plugins. We put our plugin into it. Just using your favorite utility to work with the file system. Unity loads the dynamic library only once, so if you make changes to it, you will need to restart Unity. Run and see the mirror-displayed inscription "Cam1" on the picture from the camera. This is due to the difference in texture representation in OpenGL and OpenCV.

The code is available on github in habr_part2 branch

Source: https://habr.com/ru/post/269007/

All Articles