Self-made autonomous flying machine from Android smartphone

Having fun with an autonomous aircraft is, of course, fun, but creating them yourself is even more interesting! This article is addressed to those who want to develop their own intelligent kopter and contains a set of simple instructions on how to achieve results using a smartphone on Android, OpenCV, C ++ and Java. Well, if you can go through the first steps and wish to further improve your device - at the end of the post you will find a useful link and food for thought.

Autonomous and intelligent?

In order for the kopter to fly independently, it must include all the necessary sensors, sufficient computing power and means of communication. It seems to be not so much, but almost all of the available commercial models do not have this. There are, say, models whose motion is determined by sensors located in the room. Another option is control via GPS. The GPS receiver is cheap and easy to use, but it has large delays in the receipt of data and is not accurate enough. All this does not suit us.
In order to carry the title of "intellectual" your kopter must be able to perceive and analyze the surrounding reality. This requires not only a powerful processor, a capacious battery, a high-quality camera and a sufficient set of sensors, but also fast communication devices. And of course, this whole system should be well managed and easy to program. So we come to the thought: But if not to realize the brain center of the copter based on a smartphone? The most convenient way is to use a device based on Android, since there are convenient development tools and software components for this OS, such as, for example, Intel Integrated Performance Primitives (Intel IPP) or OpenCV. A modern smartphone has all the hardware components we need, so there is absolutely no need to reinvent the wheel.

Motor control

So, the control center is selected, now it is necessary to connect motors to it. We chose the Pololu Maestro servo controller for about $ 5, it is connected via USB and in addition has a Bluetooth interface. Standard servo drives will be controlled by this card. With the help of the Pololu Maestro servo controller and the smartphone, it is relatively easy to convert a controlled aircraft into a standalone one.


')
With a few lines of code and standard Android USB tools, we will control the servo motors and, thus, the movement of the copter. A few more lines of code - and we will get access to GPS, camera and data transmission over the network.
Call the controlTransfer from UsbDeviceConnection:

import android.hardware.usb.UsbDeviceConnection; // … private UsbDeviceConnection connection; // … connection.controlTransfer(0x40, command, value, channel, null, 0, 5000); 

The controller allows you to control the servo drives, setting the end position, speed and acceleration - everything you need for a smooth movement. The command argument can take one of three values:

 public static final int USB_SET_POSITION = 0x85; public static final int USB_SET_SPEED = 0x87; public static final int USB_SET_ACCELERATION = 0x89; 

Select the appropriate values ​​and pass them to the desired servo motor using the channel argument. A link to the full source code and USB access configuration in the application manifest is provided at the end of the post.

Quadcopter features

So far, everything is going well. Hardware components are connected to each other without problems, programming is easy, because everything is implemented by means of Android. However, there is one feature associated with the design of the quadrocopter. Unlike more simple models, such as a car or a plane, the quadcopter must constantly monitor its stability. That is why its necessary component is the stabilization module. Of course, the stabilizer can be made software by writing a bunch of code in C ++ or Java. But it is much easier to buy a stabilizer card for a few dollars, which is connected directly to Pololu and controls four servo drives of the device. Everything else can be done with simple commands like ± altitude, ± speed /, ± ± inclination and ± direction.
If you create a quadcopter, keep in mind: this card will make your life a lot easier. All you need is to perform its initial calibration and then forget about it.

Putting it all together

So, as a result of the first stage of designing an autonomous quadrocopter, we have the following hardware chain:
smartphone <> micro USB-USB adapter <> USB-mini USB cable <> Pololu Maestro card <> 4 JR cables <> stabilization card <> JR cables <> servo drives <> motors



In the case of a simpler device, the chain will be shorter:
smartphone <> micro USB-USB adapter <> USB-mini USB cable <> Pololu Maestro card <> JR cables <> servo drives <> motors



In addition, you can install other drives on your flying device, for example, for flaps or landing gear. The Pololu Maestro map has management support for up to 24 drives - for our project, this is probably even superfluous.
Base platform created. Now it's time to equip our device with a vision.

Computer vision

Obviously, without a computer vision system, an intelligent device cannot be considered as such. Our kopter should be able not only to take photos, but also to analyze them - for this we use the features of OpenCV.
OpenCV is an open source library for image analysis software that underlies countless implementations of computer vision and virtual reality systems. Originally developed by Intel, it is now available for a variety of hardware platforms and operating systems.
For practice, let's try to recognize a simple sign in the form of a circle and sit in front of this sign at a certain distance. To simplify the test task, we will move the smartphone by hand.



OpenCV is not a library directly accessible to Java under Android. This is a native library, usually used from C ++ programs, so we need an Android NDK. Shooting images and visualization will be done in Java, we will use JNI for interaction between Java and C ++. We will have to install the Android NDK and Android SDK, create a new Circles project, add a C / C ++ component and change the project properties to use the OpenCV library, as shown in the screenshots below:









As a result, your project will:
The main Java file "Src / MainActivity.java"
XML markup file “Res / layout / activity_main.xml” and manifest
Two Makefiles "Jni / Android.mk" and "Jni / Application.mk"
The cpp "Jni / ComputerVision_jni.cpp" code and the "Jni / ComputerVision_jni.h" header

Unlike Java, C ++ must be compiled for a specific processor. The setting is done by editing the variable APP_ABI in the Application.mk file. If you have an Intel-based smartphone, the correct value is x86. Further NDK will make everything.

Deployment

OpenCV is a library used by an infinite number of Android applications, and the version of the library they can use is different. As a developer, you can associate your application with a specific version of OpenCV, but there is a better option. Use the dependency manager called "OpenCV Manager". This is an Android application that determines that you now need OpenCV and downloads exactly the version you need.
We want to detect circles in OpenCV, determine their center and radius, and output instructions to the smartphone operator to achieve a centered circle of the correct size. The following Java code receives the image from the camera using the Java API for Android, calls a function in C ++ via JNI, and attaches a pointer to the image in memory. C ++ code performs image processing to detect circles. Java is then called again to display the detected circles and comments.
Java code:

 … // capture images from the camera import org.opencv.Android.CameraBridgeViewBase; // load OpenCV native dependancy import org.opencv.Android.OpenCVLoader; … public void onResume() { super.onResume(); // OpenCV loading with a callback // non typical code specific to OpenCV OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_6, this, mLoaderCallback); } … // once the OpenCV manager link established, // we can load the dynamic library System.loadLibrary("jni_part"); … 

C ++ code

 … // typical for JNI : Java class method name // pointer to RGB image as argument JNIEXPORT int JNICALL Java_com_example_circles_MainActivity_process (JNIEnv *jenv, jobject obj, jlong addrGray, jlong addrRgba) … // Get the bitmap from pointer Mat& mRgb = *(Mat*)addrRgba; // blur, required before detecting circles medianBlur(mGr,mGr,5); // OpenCV detection – Hough transformation HoughCircles(mGr, //grayscale input image *circles, //output vector CV_HOUGH_GRADIENT, //detection method to use 4, //inverse ratio of the accumulator resolution to the image mGr.rows/8, //min distance between centers of detected circles 220, //higher threshold of the two passed intern canny edge detector 200, //accumulator threshold 100 20, //min radius mGr.cols/8 //max radius ); 

For the test, I moved the smartphone in front of a sheet of paper with a printed circle. For example, from a distance of 20 cm, the image of a circle will have a size of 300 pixels — we will assume this is the correct position. If the circle is smaller, the smartphone should be pushed closer, if more - then further. This is the easiest option. You can use two concentric circles, a larger one for navigation in the far distance, a smaller one - in the near range. Nothing prevents to recognize other specific shapes, such as arrows. Ultimately, we need to get a system that uses both GPS data and color information from the camera.

Future plans

Install OpenCV Manager and your APK file from Eclipse. Run it and go through all the configuration steps. It will determine the circles in the field of view and direct the movement of the smartphone to the center of the circle of a given diameter.
On the test smartphone, we received and processed a snapshot every 8 hundredths of a second - 12.5 frames per second. This proves that computer vision for a copter is a very real thing, even with limited time and financial resources.
Opportunities for further development are very wide. OpenCV is an open source library ported to many platforms. In addition, Intel IPP replaces some low-level OpenCV calls and speeds up your code by inserting functions well optimized for Intel processors. You can keep the code portability - in the future, you may need a more powerful smartphone.
Well, what to do next - you will be prompted by materials from the Intel site . It is written there in great detail how to build the flying vehicle yourself and what to teach it.
Now some more specific links:

• Original article in English in PDF
• Copter source code
• Test ARC