Some time ago I already wrote about one of my developments - the PhonoPaper technology and the program of the same name, which allows you to play sound printed in the form of a spectrogram on paper or any other surface. The process looks like this: 10 seconds of sound (voice, a piece of a song) are converted into a picture of a special format. The picture is printed and, for example, glued to the wall. A passerby, noticing the code, launches the PhonoPaper-scanner on the phone, points the camera at the picture and at the same moment begins to hear the sound encoded in it. At the same time, the user is fully involved in the process - the direction and speed of playback depend on the movement of his hand (although there is also an automatic mode). All necessary information is stored in the image, no Internet access is required.



PhonoPaper has aroused keen interest among musicians, artists, and just lovers of unusual experiments. And in the third quarter of last year, the application won first place in the Intel Rating for Developers project on Apps4All.ru. In this connection, Intel kindly provided me with a tablet based on Android x86 for further improvement and optimization of PhonoPaper. I was quick to take advantage of this, and I will tell you more about the work done and the results.
')

Video capture

The first thing that was done was the Intel INDE Media for Mobile library. Specifically, the GLCapture class for capturing video from an OpenGL ES surface in real time (in HD quality and with sound). Why do you need it? First of all, the process of searching and playing PhonoPaper codes itself is a fun, exciting spectacle that resembles playing on some unusual musical instrument. Secondly, PhonoPaper can work in a free mode when everything is converted into sound without analysis - including your carpet and a cat. Both would be great to record and share on YouTube.


Hand-drawn PhonoPaper Codes


Free mode - any image from the camera is perceived as a spectrum of sound.

The process of connecting GLCapture is repeatedly described in different articles . I'll tell you about a few points that you should know about before starting work.

• Android version must be at least 4.3. For older devices, I made a cross-platform MJPEG recorder, the speed and quality of which, of course, is much inferior to hardware-accelerated GLCapture, writing in mp4.
• The application must be built on the basis of OpenGL ES 2.0. My programs historically used version 1.1, so the code had to be rewritten. But the transition to GLES 2.0 ultimately had a positive impact on performance, since it became possible to manually configure shaders.
• GLCapture can write sound from a microphone. This is good if you need to accompany the video with your comments. If you need high-quality sound directly from the application, you will have to record it separately into a file, and then merge with mp4. For merging, you can use the MediaComposer class with the SubstituteAudioEffect effect from the Media for Mobile suite. Another way is to write to WAV, encode from WAV to AAC, add an AAC track to an mp4 file using the mp4parser library.

Since PhonoPaper is written in the Pixilang programming language, the video capture function will be further distributed to other pixilang-based applications (PixiTracker, PixiVisor, Nature - Oscillator, Virtual ANS), and most importantly - will be available to all developers using Pixilang. At the same time it is very easy to access (just a few functions: to start capturing, to stop and save).

Intel C ++ and Optimization

The next step is to build the x86 Android version of PhonoPaper using the Intel compiler (version 15.0.0) and compare the results with GCC 4.8. I am a user of Debian Linux and a rather old version to the same. Therefore, the first problem was to find the appropriate version of Intel C ++. For some reason, most of the links led to the Intel INDE project, within which there is the correct compiler, but only for Windows and OS X. This seemed strange ... Fortunately, the correct distribution was still found - this is Intel System Studio 2015 . And despite the warnings during installation, everything worked and the very first build was successful.

Compilation was performed with the following keys: -xATOM_SSSE3 -ipo -fomit-frame-pointer -fstrict-aliasing -finline-limit = 300 -ffunction-sections -restrict . To test the performance of the Pixilang virtual machine (it is based on all my applications), small tests were written, the source code and the results of which can be found in this archive . As a result, even without preliminary preparation, some pieces of code were sped up 5 (!) Times. Pretty impressive results!

In PhonoPaper most of the load goes to the function of the spectral synthesizer (table-wave, not FFT) - wavetable_generator () . A separate test was written for it, which for four seconds renders a stream of sound with a random spectrum. At the end, the test gives the highest possible sampling rate. Alas, here ICC coped worse: 105 kHz versus 100 kHz on the GCC. When compiling, we add the -qopt-report = 2 key and see the message in the report:

loop was not vectorized: vector vector.

The main loop inside our function could not be vectorized, since the input data pointers can point to overlapping sections of memory:

int* amp = (int*)amp_cont->data int* amp_delta = (int*)amp_delta_cont->data; 

As a developer, I see that in this place the intersection is excluded and you just need to inform the compiler. In C / C ++, there is a special keyword, restrict, indicating that the pointer being declared points to a block of memory that no other pointer points to. Therefore, the above code is replaced by this:

 int* restrict amp = (int*)amp_cont->data; int* restrict amp_delta = (int*)amp_delta_cont->data; 

After that, once again we collect the application and see that the cycle has been successfully vectorized. Taking into account some additional changes (in the process it turned out that it is possible to get rid of several bit operations) we have a result - 190 kHz. GCC with the same changes issued 130 kHz. We get a performance increase of 1.46 times!

What's next

As you can see, the results are very positive! PhonoPaper became faster (thanks in large part to the Intel C ++ compiler) and gained the functionality of video capture. In addition, the video recording will appear in the form of a few simple functions in the next update Pixilang 3.6.
For those who do not know, Pixilang is an open cross-platform programming language, sharpened to work with sound and graphics. The syntax of the language is very minimalistic and is a kind of a hybrid of BASIC and C, which, together with other features (the ability to write code without functions, universal containers for storing any data) reduces the threshold of entry.