Augmented reality for Dota2
I bring to your attention the harsh story about the development of a machine vision program for Android using OpenCV. The task is this - it is necessary to recognize the enemy heroes on the screen and show the victorious combination for your team.
To begin with, it was decided to develop an x86 application (C ++ / OpenCV), later porting to Android with a real camera. We needed a fast and accurate image comparison algorithm. After a long search, the most popular approaches emerged:
The remaining options are discarded as too complex or not implemented in OpenCV.
')
During the experiments, it became clear that it was necessary to first recognize the outlines of the characters, and only then compare their contents with the original icons. Search on the big screenshot for a small icon using template matching very slowly. The principle of operation is as follows:
Original image:
Mask:
Outlines of heroes:
I had to mess around with the recognition of the contours, adjust the optimal parameters for the mask and the filter of false positives. The image can be of poor quality or distorted, different proportions on different monitors.
For icon recognition, a cool bike was written that combined cv :: matchTemplate for early filtering and cv :: FlannBasedMatcher for accurate comparison. On the basis of these coefficients, the similarity weight was calculated and the hero with the highest weight returned.
Long struggled with the problem of comparing similar icons, but eventually scored, because a rare thing. For example, input image A is recognized as hero C:
Total, according to the recognition results, the error is about 10%, one frame per Core i5 is calculated up to 200ms, on Nexus7 up to 2sec. Surely you can optimize a lot of things, but for the prototype it was enough.
The main functionality of the application was ready, it remained to port it to Android, add camera support and display the result. Here I was very wrong, about 70% of the time spent working on the application was spent on fighting Android bugs on different devices.
Especially pleased camera API. Focus control, video finder resolution and photos - all this can cause problems up to a complete freeze. I advise you to test on low-end Chinese phones in order to catch maximum bugs.
It is worth mentioning that JNI is a pain. So I decided to just transfer the data to the ByteBuffer. Yes, there are all sorts of generators like SWIG, but they are terribly bulky and the exhaust code is scary.
A little advice on optimizing the size of apk. I collected all the native code into one shared library and deleted the excess, as a result, the total size was reduced by 3 times:
CFLAGS: = -DANDROID_NDK -DNDEBUG -O3 -Ofast -flto -fvisibility = hidden -ffunction-sections -fdata-sections
LDFLAGS: = -Wl, - gc-sections -Wl, - exclude-libs = ALL
Work plan
- Recognize the icons of the characters in the image of poor quality with distortion;
- Analyze the selected characters and choose a good combination for each team;
- Show result.
To begin with, it was decided to develop an x86 application (C ++ / OpenCV), later porting to Android with a real camera. We needed a fast and accurate image comparison algorithm. After a long search, the most popular approaches emerged:
- Simple euclidean distance
- Cross correlation
- Histogram comparison
- Detectors of salient points / areas
The remaining options are discarded as too complex or not implemented in OpenCV.
')
During the experiments, it became clear that it was necessary to first recognize the outlines of the characters, and only then compare their contents with the original icons. Search on the big screenshot for a small icon using template matching very slowly. The principle of operation is as follows:
- Prepare a binary mask of icons
- Find contours
- Filter out false positives
- Run recognition threads
Original image:
Mask:
Outlines of heroes:
I had to mess around with the recognition of the contours, adjust the optimal parameters for the mask and the filter of false positives. The image can be of poor quality or distorted, different proportions on different monitors.
For icon recognition, a cool bike was written that combined cv :: matchTemplate for early filtering and cv :: FlannBasedMatcher for accurate comparison. On the basis of these coefficients, the similarity weight was calculated and the hero with the highest weight returned.
Long struggled with the problem of comparing similar icons, but eventually scored, because a rare thing. For example, input image A is recognized as hero C:
Total, according to the recognition results, the error is about 10%, one frame per Core i5 is calculated up to 200ms, on Nexus7 up to 2sec. Surely you can optimize a lot of things, but for the prototype it was enough.
The main functionality of the application was ready, it remained to port it to Android, add camera support and display the result. Here I was very wrong, about 70% of the time spent working on the application was spent on fighting Android bugs on different devices.
Especially pleased camera API. Focus control, video finder resolution and photos - all this can cause problems up to a complete freeze. I advise you to test on low-end Chinese phones in order to catch maximum bugs.
It is worth mentioning that JNI is a pain. So I decided to just transfer the data to the ByteBuffer. Yes, there are all sorts of generators like SWIG, but they are terribly bulky and the exhaust code is scary.
A little advice on optimizing the size of apk. I collected all the native code into one shared library and deleted the excess, as a result, the total size was reduced by 3 times:
CFLAGS: = -DANDROID_NDK -DNDEBUG -O3 -Ofast -flto -fvisibility = hidden -ffunction-sections -fdata-sections
LDFLAGS: = -Wl, - gc-sections -Wl, - exclude-libs = ALL
Source: https://habr.com/ru/post/260353/
All Articles