Find people in photos on Android using OpenCV

Recently, I ran into one interesting puzzle for the mobile “horse” on Android — it is necessary to determine the outlines of the people in the photographs (if there were any, naturally). After searching the Internet, it was decided to use the open source project OpenCV , which can work on the Android platform.

Much has already been written about him, but this subject was not found by me and was collected from several sources and personal observations.

Customization

A small description of how to include the library in a project on Android Studio (using gradle):
To get started, you need to download the latest version of the library from the site and copy the contents of the OpenCV-2.4.8-android-sdk/sdk/java folder from the archive to the libs/OpenCV folder of your project (create if necessary).
Next we connect this module to the gradle files:
In the root project folder, edit settings.gradle and add our module:

 include ':app',':app:libs:OpenCV' 

In the build gradle file of our application (not in the root file, but app/build.gradle ) we add the line compile project(':app:libs:OpenCV') to the dependencies section in order to get:

 dependencies { compile 'com.android.support:appcompat-v7:+' compile project(':app:libs:OpenCV') } 

And create the build.gradle file in the OpenCV folder with the code:

 buildscript { repositories { mavenCentral() } dependencies { classpath 'com.android.tools.build:gradle:0.6.+' } } apply plugin: 'android-library' repositories { mavenCentral(); } android { compileSdkVersion 19 buildToolsVersion "19" defaultConfig { minSdkVersion 8 targetSdkVersion 19 } sourceSets { main { manifest.srcFile 'AndroidManifest.xml' java.srcDirs = ['src'] resources.srcDirs = ['src'] aidl.srcDirs = ['src'] renderscript.srcDirs = ['src'] res.srcDirs = ['res'] assets.srcDirs = ['assets'] } } } 

Well, as a matter of fact, that's all, the OpenCV Android SDK is connected and we can proceed to the implementation.
')

Installing libraries on the device

At the first stage of exploring OpenCV, I was confused by some feature of working under Android — the need to install the OpenCV Manager application separately, with which your creation will interact directly. Quite a strange decision, because it will be necessary to explain to the end user the fact that in order to use your application he will need to install another program in the market (the user will redirect your application directly, which will not let a person get lost, but it can still scare him away) .

There is another way to connect: static initialization , but the developers claim that it exists only for development purposes and, as it seems to me, can be removed in new versions (it’s mostly for development purposes. release package is recommended to communicate with OpenCV Manager via the async initialization described above.).

But in the context of this article, we only benefit from it. no need to mess around with connecting NDK and building / connecting sish libraries into a project. So let's continue.

To install OpenCV Manager on the emulator, we will use the adb utility from our Android SDK. To do this, start the virtual device, wait for it to load and execute the command:
/PATH_TO_ANDROID_SDK/platform-tools/adb install /PATH_TO_OPENCV/OpenCV-2.4.8-android-sdk/apk/OpenCV_2.4.8_Manager_2.16_armv7a-neon.apk (by selecting the appropriate one under the ABI apk).

Work with images

All OpenCV initialization in the application consists in implementing the BaseLoaderCallback interface callback, where there is one onManagerConnected method in which we can and can already start working with OpenCV, and calling the static OpenCVLoader.initAsync method, passing in the required parameters (including callback). If your application does not find OpenCV Manager, then ask the user to install it. Connection:

  @Override public void onResume() { super.onResume(); //    OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_8, this, mLoaderCallback); } private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) { @Override public void onManagerConnected(int status) { switch (status) { case LoaderCallbackInterface.SUCCESS: { //   OpenCV } break; default: { super.onManagerConnected(status); } break; } } }; 

Now we can safely work with our library.

In this example, we create a Bitmap from the url of the photo, then translate it into an OpenCV Mat object (image matrix), convert it from color to grayscale (this is required for the analyzer) and call the static method of the HOGDescriptor.detectMultiScale object (to which we first add a standard human detector contours from the HOGDescriptor.getDefaultPeopleDetector method). After the call, the locations variable will contain objects of rectangular areas where people are located (x, y, width, height), and in weights - search relevance (but, as practice has shown, it does not quite correspond to reality with such images).

For simplicity, I uploaded photos to facebook and combined the methods of uploading and processing photos into one. Method code itself:

 public Bitmap peopleDetect ( String path ) { Bitmap bitmap = null; float execTime; try { //   URL url = new URL( path ); HttpURLConnection connection = (HttpURLConnection) url.openConnection(); connection.setDoInput(true); connection.connect(); InputStream input = connection.getInputStream(); BitmapFactory.Options opts = new BitmapFactory.Options(); opts.inPreferredConfig = Bitmap.Config.ARGB_8888; bitmap = BitmapFactory.decodeStream(input, null, opts); long time = System.currentTimeMillis(); //     OpenCV       Mat mat = new Mat(); Utils.bitmapToMat(bitmap, mat); //    RGB    Imgproc.cvtColor(mat, mat, Imgproc.COLOR_RGB2GRAY, 4); HOGDescriptor hog = new HOGDescriptor(); //         MatOfFloat descriptors = HOGDescriptor.getDefaultPeopleDetector(); hog.setSVMDetector(descriptors); //  ,       ( locations -  , weights -  (  )  ) MatOfRect locations = new MatOfRect(); MatOfDouble weights = new MatOfDouble(); //  ,   .    locations  weights hog.detectMultiScale(mat, locations, weights); execTime = ( (float)( System.currentTimeMillis() - time ) ) / 1000f; //      Point rectPoint1 = new Point(); Point rectPoint2 = new Point(); Scalar fontColor = new Scalar(0, 0, 0); Point fontPoint = new Point(); //    -          if (locations.rows() > 0) { List<Rect> rectangles = locations.toList(); int i = 0; List<Double> weightList = weights.toList(); for (Rect rect : rectangles) { float weigh = weightList.get(i++).floatValue(); rectPoint1.x = rect.x; rectPoint1.y = rect.y; fontPoint.x = rect.x; fontPoint.y = rect.y - 4; rectPoint2.x = rect.x + rect.width; rectPoint2.y = rect.y + rect.height; final Scalar rectColor = new Scalar( 0 , 0 , 0 ); //      Core.rectangle(mat, rectPoint1, rectPoint2, rectColor, 2); Core.putText(mat, String.format("%1.2f", weigh), fontPoint, Core.FONT_HERSHEY_PLAIN, 1.5, fontColor, 2, Core.LINE_AA, false); } } fontPoint.x = 15; fontPoint.y = bitmap.getHeight() - 20; //     Core.putText(mat, "Processing time:" + execTime + " width:" + bitmap.getWidth() + " height:" + bitmap.getHeight() , fontPoint, Core.FONT_HERSHEY_PLAIN, 1.5, fontColor, 2, Core.LINE_AA, false); Utils.matToBitmap( mat , bitmap ); } catch (IOException e) { e.printStackTrace(); } return bitmap; } 

At the exit, we get a bitmap with areas of supposed finding of people superimposed on it, the weight of this search result and some additional information. The processing speed of one photo (up to a thousand pixels in width and height) on the Samsung Galaxy S3 is about 1-6 seconds. Below are search results with runtime.

In the first photo, the analyzer did not find a single person, as we would not like
Image-1.jpg width: 488 height: 420 executionTime: 1.085


Further, the result is better, but also not
Image-2.jpg width: 575 height: 400 executionTime: 1.226


Yes, and the third disappoint
Image-3.jpg width: 618 height: 920 executionTime: 6.459


Already something
Image-4.jpg width: 590 height: 505 executionTime: 3.084


Turning to a more “live” photo, the result was a bit unexpected for me.
Image-5.jpg width: 604 height: 453 executionTime: 1.913


Monument partially recognized
Image-6.jpg width: 960 height: 643 executionTime: 4.106


And here is the first photo that really shows why the library is “sharpened”
Image-7.jpg width: 960 height: 643 executionTime: 2.638


In clear contrast, did not get the desired effect.
Image-8.jpg width: 960 height: 857 executionTime: 3.293


Nothing defined here
Image-9.jpg width: 960 height: 642 executionTime: 2.264


Photography with people in the background
Image-10.jpg width: 960 height: 643 executionTime: 2.188


Close-up but without success
Image-11.jpg width: 960 height: 639 executionTime: 2.273


Instead of four, a little more.
Image-12.jpg width: 960 height: 640 executionTime: 2.669


As can be seen from the obtained results, the library is more designed to identify photos / videos from surveillance cameras, where you can select an object and confirm it with the following frames, and for photos, with previously unknown plans, gives a fairly large recognition error (you can filter by weight, but then risk losing a lot of images). The speed of image analysis does not allow using OpenCV for a large number of photos, and even when working in real time, at given powers and algorithms, it may not keep pace with the flow of frames.

For my purposes, the library did not fit, but maybe my mini research will be useful for you.

Thanks for reading!

Project on github .