Good day, Habr!

I am a member of a small company (of two people) that makes live wallpapers (live wallpapers) for Android devices. This article will cover the development of our applications, from relatively simple to more complex, applied technologies, tricks and problems solved - all with specific examples, in (almost) chronological order. All our wallpapers are fully 3D, written using OpenGL ES.
')
As a bonus - a small collection of shaders, use them as you wish. Suggestions for their improvement or correction will only be welcome - we do not claim to be a guru in this area.

“Watch” and “HTC” - the first joy

Used JPCT-AE engine, still old, which uses OpenGL ES 1.0. Loading models came from the format of 3DS. There were no shaders, compressed textures, render-to-texture, and other fashionable stuff. However, here it was possible to do without all of this.

“Christmas tree” - disadvantages of OpenGL ES 1.0

It turned out 2 things: the emulator disgusting draws three-dimensional graphics and JPCT-AE devours a lot of memory on textures.

The then emulator did not just poorly draw the three-dimensional graphics of OpenGL ES 1.0, but drew it too clearly and slowly. From now on, development has gone only on a real device.

The JPCT-AE version used at the time allocated memory for each texture, and the program could suddenly close due to lack of memory. Hopefully this bug has already been fixed in new versions of the engine. It is known that Android applications do not accept OpenGL resources in the calculation of the allocated Dalvik memory, but here the engine had problems with the non-release of memory after loading the texture into the video memory. It was necessary to reduce the size of the textures not because the video card could not cope with the drawing, but because they were all stuck in the memory and the program was falling.

Also, when using the then version of JPCT-AE, there was a nasty bug when recreating the OpenGL context - the textures could get lost or be misled. Solutions to this problem have not been found. Again, we hope that this bug has been fixed in the current version of JPCT-AE.

In fairness, we add that the latest versions of JPCT-AE have added support for OpenGL ES 2.0 and their own shaders, so that novice developers can try to use it.

“Rose” - OpenGL ES 2.0 and alpha testing

Here we switched to using pure OpenGL ES 2.0, without using any frameworks and engines.

The reason for switching to OpenGL ES 2.0 was that, to display roses in OpenGL ES 1.0, you had to use alpha-blending, with sorting polygons, of course. That all the same would lead to poor performance due to excessive redrawing (depth-testing because it turns off).

Naturally, these problems are easily solved using alpha testing, and OpenGL ES 2.0 allowed us to implement it elementary. However, there are some difficulties with this. Alpha testing itself is implemented elementarily:

vec4 base = texture2D(sTexture, vTextureCoord); gl_FragColor = base; if(base.a < 0.5) { discard; } 

However, this obvious implementation has some drawbacks related to loading a texture from a PNG file.
First, the edges of the petals get a thin black face, which is especially noticeable on the light petals:



This happens if you use the standard texture loading method, which multiplies the color by the alpha channel. Artifacts can be avoided by implementing your own texture loading.

Secondly, the texture is uncompressed, takes a lot of video memory, and therefore is drawn more slowly. Believe me, compressed textures do immediately give a performance boost. The problem here is that the only standardized texture compression algorithm for OpenGL ES 2.0 is ETC1. And he does not support the alpha channel. So you have to make two textures - for color (diffuse), and opacity. Two compressed textures still occupy less memory than one uncompressed one of the same size (and, accordingly, work faster). You can also decide that for specific cases, the transparency texture can be made smaller than the color texture, or vice versa.
But all these problems were solved later when the following live wallpapers were developed - Turbines.

“Turbines” - shaders, shaders, shaders ...

Here we began to really use the capabilities of OpenGL ES 2.0 shaders. Of the new shaders, here is the fog (still pixelated) and the landscape shader, over which the shadows from the clouds run.

The fog chose linear, although they immediately tried a more realistic exponential. The exponential fog on mobile devices slowed down so that we immediately abandoned it.

The landscape shader uses two textures - a repeating grass texture with a size of 512 * 512 and a second texture, which combines a static landscape lightmap and a cloud shadow texture (just 64 * 128). Here you can perform various operations on colors - lightmap is multiplied, shadows and fog are combined using mix ().
Then it turned out that these shaders were damn non-optimized, since everything was written in the code of the pixel shader.

“Pumpkin” - new shaders, loading acceleration

Here we implemented fast loading of models in a ready binary format, and then VBO, and for the first time applied compression of textures ETC1 (compressed textures were already mentioned earlier using the example of a rose).

It so happened that the new model loading, implemented in these live wallpapers, brought more benefits when the roses were applied on the wallpapers. Initially, the models were loaded from the OBJ format, which was rather slow and caused complaints from Roza users. After all, the model here is really rather big, a little more than 1000 triangles, and the OBJ file has been parsed for a long time. A utility was written that created data files ready for transfer to glDrawArrays (). The performance gain is difficult to describe - before the rose was loaded for 10 or more seconds, now the download can be described only as “instantly”.

I want to note that mobile devices greatly surprised us in their ability to draw a large number of polygons - as it was just said, 1000 triangles in a rose were not a problem for any device, and in our other tests and wallpapers we already use much more. But VBO was disappointing - absolutely no performance gain was noticed.

The pumpkin shader is based on some example of RenderMonkey cloth, a bit simplified and refined.

“Fading candle” - animation, simple sorting

For these wallpapers, we developed a simple vertex animation algorithm, with simple linear interpolation between frames. Software animation, i.e. on the CPU. It is used for two purposes - animation of a candle (several frames of animation) and animation of shadows. Yes, there is no shadow map here, shadows are separate models with a texture calculated in 3ds max. Model shadows have 2 frames of animation - the normal state and slightly stretched in the direction from the light source. Between these frames a smooth animation is made, synchronously with the scaling of the candle flame:



Also, the development of vertex animation allowed us to add birds to the “Turbines” wallpaper.

Well, as you can imagine, here we have been training with the blending and depth-testing switching. There are 2 transparent objects that need to be sorted - this is a candle flame and a feather. Instead of accurate sorting by distance to the camera, we check the current position of the camera. The camera moves in a circle, on a separate sector of which you need to draw first the flame, then the pen, on the rest - on the contrary.

“Lantern Festival” - vertex shaders, drawing order

As you can see from the picture, here we will talk about the shader of water and fog. What you might have missed is the fact that the sky shader also has something to tell.

The water shader is taken from the PowerVR example, and simplified (they threw out the refraction, maybe somewhere the calculations have simplified). The reflection is rendered in a small texture - only 256 * 256.

For the landscape, a rather complicated fog shader is used: the density of the fog depends on the distance and height, and the color of the fog depends on the position of the moon (the fog around the moon is highlighted). All this was possible to implement with sufficient performance only in the vertex shader. After that, we rewrote the fog shader in Wind Turbines vertex, which gave a noticeable increase in performance.

Now, about the sky shader. The sky is drawn from 3 layers: the actual sky texture with the moon, a layer of stars (stored in the alpha channel of the color texture) and moving clouds. Stars are not merged with the sky texture because they need great brightness. Initially, the shader was written something like this:

 const float CLOUDS_COVERAGE = 12.0; vec4 base = texture2D(sTexture, vTextureCoord); vec4 clouds = texture2D(sClouds, vec2(vTextureCoord.x + fOffset, vTextureCoord.y)); float stars = (1.0 - base.a) * starsCoeff; stars *= pow(1.0 - clouds.r, CLOUDS_COVERAGE); base = base + stars; gl_FragColor = base + base * clouds * 5.0; 

It would seem that the code is a bit but still the performance was low, something needed to be optimized. As you can see, to obscure the stars with clouds, the rather oily function pow () is used, and an additional enhancement of the brightness of the clouds (clouds * 5.0) is also performed. The darkening of the stars with clouds was replaced with a linear clamp (), and we managed to get rid of the gain (clouds * 5.0) altogether, making the texture of the clouds brighter in Photoshop. The final shader began to work a little faster:

 vec4 base = texture2D(sTexture, vTextureCoord); vec4 clouds = texture2D(sClouds, vec2(vTextureCoord.x + fOffset, vTextureCoord.y)); float stars = (1.0 - base.a) * starsCoeff; stars *= clamp(0.75 - clouds.r, 0.0, 1.0); base = base + stars; gl_FragColor = base + base * clouds; 

However, this performance is still not enough. Attempts to render a simplified sky into the texture of the reflection did not give anything - the texture is only 256 * 256, even if it does not draw the sky at all, nothing has changed. The problem turned out to be quite different - that the sky was painted first , not last. Those. the resources of the video map were spent on drawing the whole hemisphere of the sky, and then still wiping it with landscape and other objects. We carefully reviewed the rendering order in our program, built the objects in the optimal order and thus achieved the necessary performance. So even when it seems to you that you have completed your OpenGL application and everything is working fine, finally reconsider your rendering order - by rearranging a couple of lines of code in places, you can improve performance quite well.

“Tulip” - sorting objects

We made these live wallpapers fairly quickly - there is little modeling here, no new shaders are needed, the only thing that had to be implemented is the sorting.

Sort the need, of course, transparent "defocused" tulips. Each of them is a separate object, we calculate the square of the distance from it to the camera and sort by this value (there is enough square distance, because the value is needed only for comparison).

The performance of these wallpapers was not as big as that of Rosa, which was quite expected - many transparent polygons were added here, but they are not optimized in any way, everything that is drawn on the screen is drawn. To achieve acceptable performance, I had to pick FOV and the number of “blurry” colors on the screen - the smaller, the better.

“Jelly Bean” - glare, blending

For glare on glass and “beans,” we used simplified calculations. Usually the vector of the direction of light is involved in the glare calculations, and if we assume that it is equal to some simple constant like (0; 0; 1), then the calculations are much simpler - the dot () operation is thrown, etc. The result is obtained as if the light source is in the same place as the camera, and for such a simple scene this is more than enough.

The glass shader selects the color and alpha depending on the screen normal and flare. It works together with the correctly selected blending, naturally.

“Lotus” - dragonflies and interpolation

Let's start with shaders. The water here is the same as in the “lanterns”, only with modified parameters. For the change of day and night, multiplication by ambient-color is added to all objects:

 ... gl_FragColor *= ambientColor; 

The animation of dragonfly wings is implemented unusually - the geometry of the wings is not animated. Such a flapping of the wings is impossible to convey with the animation of geometry. The wings here are akin to the blades of a helicopter - they move very quickly and, for the naked eye, merge into a single slowly iridescent, rotating plane. So we made a model of wings from several intersecting planes, and the effect of fast swinging is created by a shader, performing a simple shift of the UV-coordinates.

Static geometry of the wings:



In these wallpapers you can observe a variety of objects moving along smooth paths - dragonflies, butterflies, fish under water and a camera. All of them move along splines, and for smooth movement, bicubic interpolation is implemented based on the Catmull-Rom algorithm. Note that interpolation is slightly different from that with which splines are smoothed in 3ds max with Smooth vertices. A hundred percent accuracy for most of the objects we did not need here. Also, the disadvantage of the applied algorithm is that for a uniform motion, the segments between the spline vertices must be of the same length - otherwise the motion on short segments will slow down, and on long segments - to accelerate. And for the movement of the camera is already important.

Note that you can use the “Normalize Spline” modifier in 3ds max to create a spline with uniform segments.

But in our case, the lack of long segments was eliminated by the way in which we exported the animation from 3ds max to a text format (in order to extract values ​​from it into an array). To export the animation, we used the “Export Animation” command, which generates an XML file (* .xaf) with animation. It is possible to find in it not only all the vertices of the curve along which the object moves, but also its position for each position on the track bar. Thus, you can animate your object as you please - with any controllers, keyframe animation, paths and even all of this at the same time, and at the output you can get its position coordinates in space depending on time.

For the halo effect of fireflies, a vertex shader was made, which positions and scales the halo sprite:

Shader Collection

The shader collection can be downloaded here: dl.dropbox.com/u/20585920/shaders.zip

All shaders are made in the program RenderMonkey, you can download it from the site AMD: developer.amd.com/resources/archive/archived-tools/gpu-tools-archive/rendermonkey-toolsuite/#download

If you are the owner of a GeForce video card, then RenderMonkey may not start. You will need to use the 'nvemulate' utility. Download here: developer.nvidia.com/nvemulate . Run it and change the setting of 'GLSL Compiler Device Support' to 'NV40':

UPD: Where to start yourself

Want to write your live wallpaper with opengl? You do not know what to do with GLWallpaperService and other cleverness? Everything has been written for a long time, take the code of examples and change it to your needs:
OpenGL ES 2.0 example from Google: code.google.com/p/gdc2011-android-opengl
OpenGL Desktop Live Wallpaper Example: github.com/markfguerra/GLWallpaperService