SoX port for Android or trying to create the perfect player
Hello, Habrayuzer!
A few months ago, I had the idea of creating a player for Android with a huge amount of effects. The only one at that time (I don’t know how it is now :), the player, with at least some sound processing, was PowerAMP, but the amount of audio effects in it was meagerly mild. I tried to implement this idea. Little has come of this, but I’ll tell you what happened in this topic. So, anyone interested, please under the cat.
For a start, I decided to find the very OpenSource library that can flexibly process sound using various effects. In the end, I accidentally met SoX . At first glance, it was an ideal library for the implementation of such a project, but it turned out that not everything is so simple ...
')
In the course of my acquaintance with SoX, I discovered the following problems:
1. The library works fine on all major operating systems, but it is not optimized for ARM.
2. The description of the library's API, as well as the API itself, was rather poor; almost all functions were easier to implement via the command line.
3. SoX, instead of using one FFmpeg for decoding, uses a number of libraries (you can look at the official site). Consequently, all these libraries also have to be collected under the Android NDK.
4. SoX perfectly reproduced and decoded almost any formats, but when applying effects, the audio could either be decoded into a file or byte array, or played through Alsa / CoreAudio. The third option did not fit unambiguously, since in theory Alsa Android is, in theory, but it does not always work and, in general, this is not the recommended method of sound reproduction. Therefore, the only option is to decode everything in the byte array and give it to Java (AudioTrack). But even this turned out to be quite difficult to realize, as ... I will explain with an example :)
Here is part of the official sound processing application example using effects:
In this case, the audio is decoded, and several effects are applied, then everything is output through ALSA. As you can see, in order to create an effect_chain, you must first have 2 threads: for reading and for displaying information. Here is the official example of decoding an audio file in parts in a byte array:
As you can see, in this case, the audio is decoded in parts, and the program receives a buffer. This example is completely working, BUT it is almost impossible to apply effects (which is the purpose of the program), because for this you would have to create a read and write stream to another buffer for each part of the audio (size 16484 bytes) and then process those streams. It did not work out for me, and in no example has this feature been described.
Then it would be logical to decode all the audio with the use of effects in one buffer, but this option uses a lot of RAM (when I tested it reached 100 MB when decoding a small audio file of 10-11 megabytes).
Despite these problems, I decided to start porting SoX and all related libraries. The week went to get acquainted with the Android NDK. In principle, porting was not a complicated, one-type process, which I described in the last article. Some libraries have already been compiled (for example, FFmpeg). In the end, I got a compiled sox.so, which really worked and is working now :) The time has come to solve problem 3. Everything turned out to be not too difficult - I created a separate stream that sends it to the Java program as the buffer fills, and she plays it. The memory problem is almost solved (after all, it is used a lot, but the application does not drop it). I got an application that reproduces almost all audio formats and adds effects to it (now a flanger, you can add any other). On the one hand, it is already possible to continue development, BUT I received two new problems at once:
1. The program uses the processor too much (reaches 80% when decoding). I do not know how to optimize C code under ARM, so I do not see it possible to solve this myself.
2. The program is unstable. Approximately 1-2 of 10 launches it does not play anything (tested on the Galaxy Tab). I also could not solve this problem ...
Total, so that the work is not lost, I decided to publish all the code on GitHub and write an article here. Link to code. If someone sees an opportunity to solve these problems - write me in email or in lichku. I hope my idea will interest someone :)
A few months ago, I had the idea of creating a player for Android with a huge amount of effects. The only one at that time (I don’t know how it is now :), the player, with at least some sound processing, was PowerAMP, but the amount of audio effects in it was meagerly mild. I tried to implement this idea. Little has come of this, but I’ll tell you what happened in this topic. So, anyone interested, please under the cat.
Looking for a library ...
For a start, I decided to find the very OpenSource library that can flexibly process sound using various effects. In the end, I accidentally met SoX . At first glance, it was an ideal library for the implementation of such a project, but it turned out that not everything is so simple ...
')
First problems
In the course of my acquaintance with SoX, I discovered the following problems:
1. The library works fine on all major operating systems, but it is not optimized for ARM.
2. The description of the library's API, as well as the API itself, was rather poor; almost all functions were easier to implement via the command line.
3. SoX, instead of using one FFmpeg for decoding, uses a number of libraries (you can look at the official site). Consequently, all these libraries also have to be collected under the Android NDK.
4. SoX perfectly reproduced and decoded almost any formats, but when applying effects, the audio could either be decoded into a file or byte array, or played through Alsa / CoreAudio. The third option did not fit unambiguously, since in theory Alsa Android is, in theory, but it does not always work and, in general, this is not the recommended method of sound reproduction. Therefore, the only option is to decode everything in the byte array and give it to Java (AudioTrack). But even this turned out to be quite difficult to realize, as ... I will explain with an example :)
Here is part of the official sound processing application example using effects:
assert(sox_init() == SOX_SUCCESS); assert(in = sox_open_read(argv[1], NULL, NULL, NULL)); /* Change "alsa" in this line to use an alternative audio device driver: */ assert(out= sox_open_write("default", &in->signal, NULL, "alsa", NULL, NULL)); chain = sox_create_effects_chain(&in->encoding, &out->encoding); e = sox_create_effect(sox_find_effect("input")); args[0] = (char *)in, assert(sox_effect_options(e, 1, args) == SOX_SUCCESS); assert(sox_add_effect(chain, e, &in->signal, &in->signal) == SOX_SUCCESS); e = sox_create_effect(sox_find_effect("trim")); args[0] = "10", assert(sox_effect_options(e, 1, args) == SOX_SUCCESS); assert(sox_add_effect(chain, e, &in->signal, &in->signal) == SOX_SUCCESS); if (in->signal.rate != out->signal.rate) { e = sox_create_effect(sox_find_effect("rate")); assert(sox_effect_options(e, 0, NULL) == SOX_SUCCESS); assert(sox_add_effect(chain, e, &in->signal, &out->signal) == SOX_SUCCESS); } if (in->signal.channels != out->signal.channels) { e = sox_create_effect(sox_find_effect("channels")); assert(sox_effect_options(e, 0, NULL) == SOX_SUCCESS); assert(sox_add_effect(chain, e, &in->signal, &out->signal) == SOX_SUCCESS); } e = sox_create_effect(sox_find_effect("output")); args[0] = (char *)out, assert(sox_effect_options(e, 1, args) == SOX_SUCCESS); assert(sox_add_effect(chain, e, &in->signal, &out->signal) == SOX_SUCCESS); sox_flow_effects(chain, NULL, NULL); In this case, the audio is decoded, and several effects are applied, then everything is output through ALSA. As you can see, in order to create an effect_chain, you must first have 2 threads: for reading and for displaying information. Here is the official example of decoding an audio file in parts in a byte array:
assert(sox_init() == SOX_SUCCESS); /* Open the input file (with default parameters) */ assert(in = sox_open_read(argv[1], NULL, NULL, NULL)); #if defined FIXED_BUFFER assert(out = sox_open_mem_write(buffer, buffer_size, &in->signal, NULL, "sox", NULL)); #else assert(out = sox_open_memstream_write(&buffer, &buffer_size, &in->signal, NULL, "sox", NULL)); #endif while ((number_read = sox_read(in, samples, MAX_SAMPLES))) assert(sox_write(out, samples, number_read) == number_read); As you can see, in this case, the audio is decoded in parts, and the program receives a buffer. This example is completely working, BUT it is almost impossible to apply effects (which is the purpose of the program), because for this you would have to create a read and write stream to another buffer for each part of the audio (size 16484 bytes) and then process those streams. It did not work out for me, and in no example has this feature been described.
Then it would be logical to decode all the audio with the use of effects in one buffer, but this option uses a lot of RAM (when I tested it reached 100 MB when decoding a small audio file of 10-11 megabytes).
But still decided to try ...
Despite these problems, I decided to start porting SoX and all related libraries. The week went to get acquainted with the Android NDK. In principle, porting was not a complicated, one-type process, which I described in the last article. Some libraries have already been compiled (for example, FFmpeg). In the end, I got a compiled sox.so, which really worked and is working now :) The time has come to solve problem 3. Everything turned out to be not too difficult - I created a separate stream that sends it to the Java program as the buffer fills, and she plays it. The memory problem is almost solved (after all, it is used a lot, but the application does not drop it). I got an application that reproduces almost all audio formats and adds effects to it (now a flanger, you can add any other). On the one hand, it is already possible to continue development, BUT I received two new problems at once:
1. The program uses the processor too much (reaches 80% when decoding). I do not know how to optimize C code under ARM, so I do not see it possible to solve this myself.
2. The program is unstable. Approximately 1-2 of 10 launches it does not play anything (tested on the Galaxy Tab). I also could not solve this problem ...
Total, so that the work is not lost, I decided to publish all the code on GitHub and write an article here. Link to code. If someone sees an opportunity to solve these problems - write me in email or in lichku. I hope my idea will interest someone :)
Source: https://habr.com/ru/post/131377/
All Articles