Android NDK: OpenSL ES

Hello, dear habrazhiteli!
Recently, reading Habr, I saw an article on the Android NDK and OpenAL. And in the comments a question was asked about OpenSL ES. Then I had the idea to write an article about this library. I dealt with this topic when I needed to add sounds and music to the game for Android, written in C ++, under the NDK. The article does not claim to be complete, there will be only the basics.

Content:

1. A brief description of the structures of OpenSL ES
2. Initializing the library mechanism and creating an object to work with speakers
3. PCM playback (wav)
4. MP3 playback, OGG
5. Conclusion

1. Brief description of structures

Working with OpenSL ES is based on pseudo-object-oriented structures of the C language. They are used when the project is written in C, but I want object-oriented. In general, pseudo-object-oriented structures are the usual C structures, containing pointers to functions that, with the first argument, get pointers to the structure itself, like this in C ++, but explicitly.
In OpenSL ES, there are two main types of structures described above:

• An object (SLObjectItf) is an abstraction of a set of resources designed to perform a specific set of tasks and to store information about these resources. When creating an object, its type is defined, which determines the range of tasks that can be solved with its help. An object resembling an Object of the Java language can be considered a similarity of a class in C ++.
• Interface (SLEngineItf, SLPlayItf, SLSeekItf, etc.) is an abstraction of a set of interrelated functionality provided by a specific object. The interface includes a variety of methods used to perform actions on an object. The interface has a type that defines the exact list of methods supported by this interface. An interface is defined by its identifier, which can be used in the code to refer to an interface type.

Simply put, objects are needed to allocate resources and get interfaces, and interfaces provide access to the capabilities of objects. A single object can have multiple interfaces. Depending on the device, some interfaces may not be available. However, I did not come across this.
')

2. Initializing the library mechanism and creating an object for working with speakers

To connect OpenSL ES to the Android NDK, simply add the lOpenSLES flag to the LOCAL_LDLIBS section of the Android.mk file:

LOCAL_LDLIBS := /*...*/ -lOpenSLES 

Headers used:

 #include <SLES/OpenSLES.h> #include <SLES/OpenSLES_Android.h> 

To start working with OpenSL ES, you must initialize an object of the OpenSL ES mechanism (SLObjectItf) by calling slCreateEngine, specifying that you will use the SL_IID_ENGINE interface to work with it. This is necessary in order to be able to create other objects. The object obtained by such a call becomes the central object for accessing the OpenSL ES API. Next, the object must be implemented using the Realize pseudo method, which is analogous to the constructor in C ++. The first parameter of Realize is the object itself (an analogue of this), and the second is the async flag indicating whether the object will be asynchronous.
The current implementation of the Android NDK makes it possible to create only one library mechanism and up to 32 objects in general. However, any object creation operation may fail (for example, due to a lack of system resources).

Next, you need to get the interface SL_IID_ENGINE, through which we will have access to the speakers, playing music, sounds, and so on.

Let us dwell on the general scheme of working with objects:

• Get object by specifying desired interfaces.
• Implement it by calling (*obj)->Realize(obj, async);
• Get the required interfaces by calling (*obj)-> GetInterface (obj, ID, &itf);
• Work with interfaces
• Delete an object by calling (*obj)->Destroy(obj);

To work with speakers, create an outputMixObj object using the CreateOutputMix pseudo method of the engineObj engine interface (This just sounds scary so that the reader learns to distinguish between objects and interfaces). We will need this object later for sound output.

SLOutputMixItf is an object representing a sound output device (speaker, headphone). The OpenSL ES specification provides for the possibility of obtaining a list of available I / O devices, but the Android NDK implementation is not complete enough and does not support either getting the list of devices or choosing what you want (the SLAudioIODeviceCapabilitiesItf interface is officially intended for this).

3. Playing PCM (wav)

At once I will make a reservation that for simplification I do not use the data from the WAV header. If desired, add support for this is fairly easy. Here, the header is needed only to correctly determine the size of the data.

Now let's set up a quick buffer sound output. To do this, use the specialized extension SLDataLocator_AndroidSimpleBufferQueue. Also, to play music, you must fill in two structures: SLDataSource and SLDataSink, which describe the input and output of the audio channel, respectively.

In addition to SL_IID_PLAY and SL_IID_BUFFERQUEUE, you can request other interfaces, for example:

• SL_IID_VOLUME for volume control
• SL_IID_MUTESOLO for managing channels (for multichannel audio only, this is indicated in the numChannels field of the SLDataFormat_PCM structure).
• SL_IID_EFFECTSEND for applying effects (by specification - only reverb effect)

etc.
Call (*player)->SetPlayState(player, SL_PLAYSTATE_PLAYING); we turn on the newly created player. While the queue is empty, so you can only hear silence. Let's put some sound in the queue.

That's it, the simplest wav player is ready.
It should be noted that in peak specification, the Android NDK does not support buffered output of music in formats other than PCM.

4. Playing MP3, OGG

The scheme described above is not suitable for playing long music files. First of all, due to the fact that a long wav file will weigh very, very much. It is better to use MP3 or OGG. OpenSL ES supports streaming files out of the box. The difference from the buffered output is also in the fact that for each music file you need to create a separate player object. It is impossible to change the file during playback for this player.
Prepare to play music:


Next, re-fill the SLDataSource and SLDataSink. And create an audio player.

To describe the source data we use the MIME type, it provides automatic detection of the file type.
Next we get the interfaces SL_IID_PLAY and SL_IID_SEEK. The latter is needed to change the playback position in the file and looping. It can be used regardless of the playback status and speed.

In theory, the looping mechanism should be convenient for setting background music in the game. In practice, between the end of the composition and its beginning 0.5-1.0 seconds passes (time for hearing, floats on different devices). I overcame this by making some smooth fading in the middle and end in the background music. So the gap is not noticeable.
According to the specification, various callbacks can be attached to the SLPlayItf interface. The Android NDK feature is not supported (the method returns SL_RESULT_SUCCESS, but callbacks do not work).
To stop or pause the player, you can use the SetPlayState method of the SLPlayItf interface with the SL_PLAYSTATE_STOPPED or SL_PLAYSTATE_PAUSED values, respectively. GetPlayState method, which returns the same values, allows you to find out the status of the player.

5. Conclusion

The OpenSL ES API is quite rich, and besides sound reproduction, it allows recording it. Here I will not touch the sound recording, I will only say that it exists and works quite well. To get the data, a queue of buffers is used. Data comes in PCM format.
The library is difficult to use in cross-platform development, because Many features are implemented using Android-specific methods. Nevertheless, it seemed to me quite comfortable.
The cons seems to be a free implementation; many things from the specification are not supported. In addition, this API is not faster than the API available in the Android SDK.

Literature

1. Sylvain Retabouil. Android NDK. Development of applications for Android on C / C ++.
2. The Khronos Group Inc. OpenSL ES Specification .

Good and more complete code samples can be found in the standard Android NDK (NativeAudio project).
Anticipating questions about the need to use the Android NDK in general and OpenSL ES in particular, I will answer immediately. Android NDK was needed on the condition of the test task from a well-known game development company (there were contests for Habré). Later it turned into a challenge to me: can I finish the job well. Smog. OpenSL ES chose on a whim, because I had no experience with him or with OpenAL, but for this I considered calls to Java to be an ugly solution.