As we in Smart Engines taught Sailfish OS recognition

Hello! As you already know from our articles, we in Smart Engines are engaged in recognition, and we try to recognize on anything and in any conditions. We support all popular operating systems: iOS, Android, Windows, Linux, MacOS, Solaris. We also support domestic producers: Elbrus and AstraLinux. Our algorithms are optimized for ARMv7-v8, AArch64, x86, x86_64, SPARC, E2K, MIPS.

Therefore, when we saw the growing popularity of the Russian operating system Sailfish Mobile OS RUS, we could not avoid it. Sailfish Mobile OS RUS is a POSIX-compatible operating system for mobile devices, developed by the domestic company Open Mobile Platform for solving problems of corporate users and government agencies. As of February 2018, it is the only mobile operating system included in the register of Russian software and certified by the Federal Security Service of the AK1 / KC1 class.

In this article we will tell you about our experience of porting our Smart IDReader ( Hieroglyph technology) to our recognition library on Sailfish OS. It will contain code, links and videos. We want this article to be technically informative and useful as a general instruction for those who port C ++ applications to Sailfish OS.

We are grateful to colleagues from the Open Mobile Platform (OMP), Kirill Chuvilin and Alexey Andreev, for consultations in developing the application and providing two INOI R7 devices for testing and demonstration.

TL; DR - porting was quick and easy. In total, it took less than one working man-week from downloading Sailfish OS SDK to removing the first video with a demonstration of document recognition on the INOI R7 device (the video will be at the end of the article).

Installation toolkit

The first step was to install the Sailfish OS SDK. You can download the installation package from the official site , but we chose an even simpler method and installed the SDK from AUR, the Arch Linux user repository (the most popular Linux distribution among Smart Engines developers), in which, as usual, everything is there and installed in one command:

yaourt -S sailfishos-sdk-bin 

Development tools include three main components:

1. Environment and build system - Oracle VM VirtualBox image containing installed compilers and libraries for building applications for a device or emulator.
2. SailfishOS Emulator, similar to emulators for other mobile platforms (iOS, Android, ...)
3. IDE is a well-known Qt Creator, completed to support interaction with the build system and launch the program on a device or emulator.

The connection to the virtual machine for building occurs via SSH. The only problem was that when installing the SDK for the virtual machine and IDE, port 2222 hard-coded, which was already used for other purposes on the developer's computer. In VirtualBox, this port changes in an understandable way, but in the IDE it was impossible to change it from the GUI. Fortunately, it could be changed in the local settings mersdk.xml file:

 <!-- ~/.config/SailfishOS-SDK/qtcreator/mersdk.xml --> <value type="QString" key="SshPort">2222</value> 

After solving this problem, Cyril was informed about this problem, and there were no further problems with the toolkit.

Build C ++ Core Recognition

Our recognition engine is written in C ++, and in this case, the obvious choice for porting to Sailfish OS was to build the kernel in the form of static libraries.

A short but informative manual for assembling libraries is on the Building Sailfish OS packages manually pages of official documentation. By it, it became clear that you just need to connect via SSH to a virtual machine, and then call the usual build commands indicating the correct architecture. We describe the process in steps.

1. We start the Sailfish OS Build Engine virtual machine (rather headless mode)

2. Connect to it:

   
   

    ssh -p 2222 -i /opt/SailfishOS/SDK/vmshare/ssh/private_keys/engine/mersdk mersdk@localhost 

   
   

   By the way, the default external user home directory is available in the virtual machine at /home/src1 .

   
   

3. We get a list of supported platforms:

   
   

    [mersdk@SailfishSDK ~]$ sdk-assistant list Toolings: SailfishOS-2.1.3.7 Targets: SailfishOS-2.1.3.7-armv7hl #  SailfishOS-2.1.3.7-i486 #  

   
   

   Unlike the example on the site, in the names of the platforms there may be a version number that must also be taken into account. Further, the assembly will be carried out for the device (SailfishOS-2.1.3.7-armv7hl), but for the emulator the whole process is similar.

   
   

   For the required architecture, the corresponding environment is provided by the sb2 (Scratchbox2) command, so everything needs to be done through it:

   
   

    sb2 -t SailfishOS-2.1.3.7-armv7hl <args....> 

   
   

4. Install the necessary packages for the assembly. In our case, these were CMake, GCC, G ++ and Zip:

   
   

    sb2 -t SailfishOS-2.1.3.7-armv7hl -m sdk-install -R zypper in cmake gcc gcc-c++ zip 

   
   

5. Create a build folder and call CMake , and pass it the correct CMAKE_SYSROOT so that the compilers are searched for the right place for us:

   
   

    mkdir build && cd build sb2 -t SailfishOS-2.1.3.7-armv7hl -m sdk-build cmake -DCMAKE_SYSROOT=/srv/mer/targets/SailfishOS-2.1.3.7-armv7hl .. 

   
   

6. We collect the project:

   
   

    sb2 -t SailfishOS-2.1.3.7-armv7hl -m sdk-build make install -j8 

   
   

That's all! Thus, the build under Sailfish OS is practically no different from the build under Linux - you just need to connect to the virtual machine and invoke commands with the correct environment.

Creating and configuring a Qt project

The process of creating a project is trivial and you can read about it, for example, in the Sailfish OS hub in Habrahabr. After that, it was necessary to do only one thing - add paths to header files and static libraries in the .pro file as follows:

 INCLUDEPATH += /path/to/install.armv7-linux.release/include LIBS += -L/path/to/install.armv7-linux.release/lib \ -lsomeStaticLibrary \ -lanotherStaticLibrary 

After that, the project with the call of our C ++ recognition kernel was successfully compiled and launched.

Writing GUI with QML and Sailfish Silica

GUI applications for Sailfish OS are written using Qt in C ++ and QML: both with the standard QtQuick library and the special Sailfish SIlica .

Almost all the required information on these technologies is present in their official documentation. Additional useful links include the Sailfish OS development hub on Habré, cribs for colors and indents , icons and main components , as well as a free introductory course for Stepik.

To draw graphic primitives, we used the Canvas and its onPaint method. Selection of a document by horizontal scrolling - SlideshowView (thanks to Alexey for the implementation example). Display recognition results - SilicaListView with ListModel.

In addition to writing basic GUI elements, it is important to learn how to call C ++ classes from QML code, which is quite simply and clearly described on the Integrating QML and C ++ page of Qt official documentation: we make C ++ class, register it in C ++, import it into QML. For wrapping C ++ structures in QML, the most obvious way was to use Q_GADGET , about which there is even an article on Habré.

Interaction with the camera device: a detective-scientific investigation

Understanding how to properly interact with the camera API was the only difficulty in the whole process of developing a demo application and deserves a separate point - it took a little research with brainstorming, hypotheses and experiments.

The scenario of using our real-time video stream recognition libraries is usually this: as long as the user sees the camera preview and holds an object in front of it (for example, a Russian passport or bank card), the frames coming from the camera are recognized one by one and recognized by the recognition library. At the same time, during the recognition of each frame, the user is shown intermediate information about the location of the document, its fields, etc. Thus, it is required to simultaneously capture frames from the camera and show previews to the user.

When creating the camera and the preview in QML, there were no questions: create a Camera and refer to it in VideoOutput :

 Camera { id: camera position: Camera.BackFace // ... } VideoOutput { anchors.fill: parent source: camera } 

It remains to find a way to intercept the frames coming from the camera and transfer them to C ++ for recognition.

As a first method, we tried the standard QVideoProbe , which exists specifically for receiving frames coming from the camera. However, the call to probe.setSource(camera) returned false , after which nothing came to the callback to intercept frames. Judging by the information on the Internet, this problem is also relevant for Android devices, which may indicate a simple lack of implementation of QVideoProbe for Sailfish OS. We hope that soon it will be fixed.

The second method is the inheritance from QAbstractVideoSurface and the implementation of the present(const QVideoFrame &frame) method present(const QVideoFrame &frame) , which passes the present(const QVideoFrame &frame) frame to recognition. After calling camera.setViewFinder(&videoSurface) frames really began to arrive (in NV21 format) and even be recognized if you hold the document in front of the camera ... that's just the preview in VideoOutput was lost and the user lost the opportunity to see what his camera was shooting. It was not possible to quickly defeat this problem, and we, along with Kirill and Alexey from the OMP, began to look for a third alternative.

Fortunately, the third way of collective effort was found fairly quickly. It turned out to be a bunch of QAbstractVideoFilter , which by createFilterRunnable() method creates QVideoFilterRunnable , which in turn implements the frame processing functionality. After that, the implemented filter should be added to QML and referenced in the existing VideoOutput:

 RecognitionVideoFilter { id: recognitionVideoFilter // ... } VideoOutput { anchors.fill: parent source: camera filters: [ recognitionVideoFilter ] } 

Compiled, run. There are video previews, but still no frames. And suddenly, Cyril shared insider information that for the operation of the filters on the device, you need to update the QtMultimedia plugin, which turned out to be available via the link . After that, everything worked - frames in the BGRA format started coming to the core. A small detail - the QAbstractVideoSurface::run function blocks the camera preview (which is logical, because the filter involves changing the frame, and we only need to read and recognize it), so for a smooth preview it was necessary to run the recognition in a separate stream - for example, through QtConcurrent :: run (...) .

That's all! It only remained to add the GUI of the application, which was not difficult.

Demonstration at the Mobile World Congress 2018 in Barcelona and other videos

This year, our company presented its recognition technology for the second time at the annual Mobile World Congress 2018 in Barcelona, ​​which again impressed us with its scale: more than 107,000 visitors from 205 countries, 2400 stands of companies located on 120,000 square meters of pavilions. Like last year , it made a very positive impression on us.

Of course, we did not miss the opportunity to go bragging our demo at the booth of the Finnish company Jolla, the original developer of Sailfish OS, and to show the work of the program "live" to colleagues from the Open Mobile Platform, who even shot several videos and posted them on VKontakte :

• First video
• Second video
• Third video

But the video from the first working version of the program:

Many other videos with demonstrations of Smart Engines technology can be viewed on our YouTube channel .

Conclusion

In this article, we talked about our experience of porting the Smart IDReader recognition library to Sailfish OS, including writing a demo application. We were pleasantly surprised by the amount of relevant official documentation and the quality of the toolkit. Of course, a couple of technical issues arose, but they managed to win quickly. We hope that our experience will be useful to others.