Creating dials for Android Wear on OpenGL ES 2.0
As soon as Google announced the possibility of creating custom dials in the new Android 5.0 for Android Wear, we ordered a new ASUS ZenWatch in the nearest online store in order to try out this latest feature. It was decided not to port one of the existing three-dimensional live wallpapers, but to create a new scene just for the clock. As a result, a concept was created and an application with a set of five digital dials was created, implemented in 3D using OpengGL ES 2.0.
The idea of ​​the dial originated under the impression of this work (by beeple):
')
The video looks cool and does not seem complicated at first glance. However, we found some technical limitations that prevented the creation of the same scene as in the video. It’s not so easy to make numbers from lines (wireframe) that would correctly overlap each other. In short, this would require a large number of draw commands (draw calls) and a very limited clock iron could not cope with so many draw calls. So we moved to another idea - add depths to the numbers from the pixel fonts from old computers.
The Gles2WatchFaceService class from the Android Wear API does not provide access to all the features necessary to create a full-fledged three-dimensional dial scene. The main problem that we immediately encountered was the impossibility of choosing the necessary configuration of OpenGL ES. In fact, this class does not provide access to EGL at all. It is enough to start the 'Tilt' dial from the API example, but it has no flexibility to customize EGL. This example does not require a depth buffer, as it does not contain any intersecting geometry. In Gles2WatchFaceService , the configuration is simply selected without EGL_DEPTH_SIZE and the API does not allow it to be changed.
In view of the discovery of such a significant limitation, it was decided to decompile the source code and create its own Gles2WatchFaceService implementation with blackjack and the correct EGL configuration.
Also, a very well-known developer of live wallpaper and dials Kittehface Software reported on his blog that Moto360 does not work with a completely correct 16-bit EGL configuration. Therefore, on all devices, our application always uses 32-bit color. Many thanks to Kittehface Software for warning us and other developers about this - finding the cause of the crash on Moto360 could take days, and we didn’t have the clock itself at that time.
We will not provide the full code of the decompiled class Gles2WatchFaceService because it will still change with the release of the new API. Here are the changes we made in the decompiled Gles2WatchFaceService :
1. Update the search for a suitable config with the depth component:
2. Make the mInsetBottom and mInsetLeft variables available from child classes. They will be used to correctly update the viewport. For example, we make them protected :
Official documentation has very scanty and vague references to how to work with devices with different screens using the onApplyWindowInsets () method. It says that this method should be used in order to adapt to screens with a cut-off bottom edge. This is necessary in order to adjust the view on devices such as the Moto 360.
Having no Moto360 in hand, it was not clear how to use this method so that our first attempts to launch the application on this watch resulted in the view being shifted - shifted upward by the size of the cropped portion of the screen. It was rather strange that this problem was not in the example of the Tilt dial - it was correctly centered. To understand this, I had to look again into the decompiled code of the Gles2WatchFaceService service. The reason was the use of glViewport () . To implement the glow effect (bloom), we used a render to texture and called glViewport () to switch rendering to texture or screen. In order to avoid shifting the image, you need to take into account the value of the lower indent for cropped screens when you set glViewport () .
For unknown reasons, it is impossible to draw opaque alert cards (peek cards) in screen saver mode (ambient mode). In normal mode, you can specify the type of cards, but in economy mode they are always completely transparent. You will have to draw the black rectangle yourself in the place where the card will be displayed above it using the getPeekCardPosition () method. In our case, it was enough to use small-sized cards and they did not overlap with the numbers, but in general it is desirable to reduce the size of the drawing area in order according to the size of the card.
In our implementation of the dial setting application, we use the HoloColorPicker to select numbers and background colors. The color change on the clock occurs in real time as soon as the user changes the color (in the ColorPicker.OnColorChangedListener event of the color selection control). However, this causes some problems. The user can change color very quickly (an event occurs when a finger moves along the “ring” of color selection), which leads to an overflow of the queue of messages about color change. The Wearable Data Layer API is not designed for sending messages so intensely. Nothing falls, but some of the recent messages may simply not reach the hours. To avoid this, we restrict sending color updates to an interval of 500 ms. Such an insignificant delay does not cause any inconvenience and allows us not to overload the capabilities of the Wearable Data Layer API:
Most of the watches for Android Wear are based on the Snapdragon 400 chip - ASUS, Sony, Samsung and LG use it in all their watches. This chip has an impressive quad-core processor with a frequency of up to 1.2 Hz, which is more than enough for a clock. Its Adreno 305 video card may seem a bit outdated at first glance. But the clock has a rather small resolution of 320x320 pixels, so its power is enough to draw quite complex 3D scenes at 60 frames per second. Even the Moto 360 with its outdated OMAP3630 chip has a fairly powerful PowerVR SGX530 video card, which delivers sufficient performance at the available screen resolution.
For example, the additional effect of the bloom in the resolution of 128x128 with a four-pass blur did not cause any drop in performance - Adreno 305 easily coped with this additional task. However, it was enough for the clock to use a lower resolution of a 64x64 bloom with two blur cycles.
There are a lot of videos where people show games such as Temple Run 2 or GTA running on an Android clock that work without lag - this once again demonstrates the performance of the video cards in these devices.
To reduce the number of drawing commands and the work done by the CPU, the figures are animated in a vertex shader. To illustrate this, here is an example of a transition between the numbers “5” and “0”:
As you can see, the model vertices are divided into 3 groups:
This information is entered in the texture coordinates - the V coordinate is set to 0 for fixed parts, 1 for those that go up, and -1 for those that go down. In addition to this, the U-coordinate sets the phase of the animation, so that the cylinders do not move all together, but with a slight lag from each other:
The fragment shader is very simple - the smaller the Z-coordinate of the vertex, the more black the final pixel is drawn. Here’s what the transition between the numbers “3” and “4” looks like (models are mirrored due to RenderMonkey) - you can see how the parts from “4” are shifted down, and the parts “3” are lifted up:
Here is the final result when the bottom of the model merges with the black background:
uHeight sets the height of the animation - the value depends on the “font” and sets the height of a separate column of the model - equal to 40 units for the font shown in the pictures.
uHeightColor and uHeightOffset allow you to vary the color transition to black. The values uHeightColor = 40 and uHeightOffset = 0 were used for the examples in the pictures, but they are different for different fonts.
Link to the archive with shaders for the RenderMonkey program: dl.dropboxusercontent.com/u/20585920/shaders_watchface.rar
When you publish an Android Wear app on Google Play, you should check the “Distribute your app on Android Wear” box. This initiates the process of moderating the application for compliance with the Wear App Quality requirements. In our case, this process took only a few hours.
Initial concept
The idea of ​​the dial originated under the impression of this work (by beeple):
')
The video looks cool and does not seem complicated at first glance. However, we found some technical limitations that prevented the creation of the same scene as in the video. It’s not so easy to make numbers from lines (wireframe) that would correctly overlap each other. In short, this would require a large number of draw commands (draw calls) and a very limited clock iron could not cope with so many draw calls. So we moved to another idea - add depths to the numbers from the pixel fonts from old computers.
OpenGL ES configuration without depth component
The Gles2WatchFaceService class from the Android Wear API does not provide access to all the features necessary to create a full-fledged three-dimensional dial scene. The main problem that we immediately encountered was the impossibility of choosing the necessary configuration of OpenGL ES. In fact, this class does not provide access to EGL at all. It is enough to start the 'Tilt' dial from the API example, but it has no flexibility to customize EGL. This example does not require a depth buffer, as it does not contain any intersecting geometry. In Gles2WatchFaceService , the configuration is simply selected without EGL_DEPTH_SIZE and the API does not allow it to be changed.
In view of the discovery of such a significant limitation, it was decided to decompile the source code and create its own Gles2WatchFaceService implementation with blackjack and the correct EGL configuration.
Also, a very well-known developer of live wallpaper and dials Kittehface Software reported on his blog that Moto360 does not work with a completely correct 16-bit EGL configuration. Therefore, on all devices, our application always uses 32-bit color. Many thanks to Kittehface Software for warning us and other developers about this - finding the cause of the crash on Moto360 could take days, and we didn’t have the clock itself at that time.
We will not provide the full code of the decompiled class Gles2WatchFaceService because it will still change with the release of the new API. Here are the changes we made in the decompiled Gles2WatchFaceService :
1. Update the search for a suitable config with the depth component:
private static final int [] EGL_CONFIG_ATTRIB_LIST = new int [] {
EGL14.EGL_RENDERABLE_TYPE, 4,
EGL14.EGL_RED_SIZE, 8,
EGL14.EGL_GREEN_SIZE, 8,
EGL14.EGL_BLUE_SIZE, 8,
EGL14.EGL_ALPHA_SIZE, 8,
EGL14.EGL_DEPTH_SIZE, 16, // this was not enough
EGL14.EGL_NONE};
2. Make the mInsetBottom and mInsetLeft variables available from child classes. They will be used to correctly update the viewport. For example, we make them protected :
protected int mInsetBottom = 0; protected int mInsetLeft = 0;
glViewport ()
Official documentation has very scanty and vague references to how to work with devices with different screens using the onApplyWindowInsets () method. It says that this method should be used in order to adapt to screens with a cut-off bottom edge. This is necessary in order to adjust the view on devices such as the Moto 360.
Having no Moto360 in hand, it was not clear how to use this method so that our first attempts to launch the application on this watch resulted in the view being shifted - shifted upward by the size of the cropped portion of the screen. It was rather strange that this problem was not in the example of the Tilt dial - it was correctly centered. To understand this, I had to look again into the decompiled code of the Gles2WatchFaceService service. The reason was the use of glViewport () . To implement the glow effect (bloom), we used a render to texture and called glViewport () to switch rendering to texture or screen. In order to avoid shifting the image, you need to take into account the value of the lower indent for cropped screens when you set glViewport () .
Transparency in normal and ambient mode
For unknown reasons, it is impossible to draw opaque alert cards (peek cards) in screen saver mode (ambient mode). In normal mode, you can specify the type of cards, but in economy mode they are always completely transparent. You will have to draw the black rectangle yourself in the place where the card will be displayed above it using the getPeekCardPosition () method. In our case, it was enough to use small-sized cards and they did not overlap with the numbers, but in general it is desirable to reduce the size of the drawing area in order according to the size of the card.
Sending messages with settings for updating the clock face
In our implementation of the dial setting application, we use the HoloColorPicker to select numbers and background colors. The color change on the clock occurs in real time as soon as the user changes the color (in the ColorPicker.OnColorChangedListener event of the color selection control). However, this causes some problems. The user can change color very quickly (an event occurs when a finger moves along the “ring” of color selection), which leads to an overflow of the queue of messages about color change. The Wearable Data Layer API is not designed for sending messages so intensely. Nothing falls, but some of the recent messages may simply not reach the hours. To avoid this, we restrict sending color updates to an interval of 500 ms. Such an insignificant delay does not cause any inconvenience and allows us not to overload the capabilities of the Wearable Data Layer API:
Handler handlerUpdateColorBackground = new Handler ();
Runnable runnableUpdateColorBackground = new Runnable () {
@Override
public void run () {
// update only if color was changed
if (pickedColorBackground! = lastSentColorBackground) {
sendConfigUpdateMessage (KEY_BACKGROUND_COLOR, pickedColorBackground);
}
handlerUpdateColorBackground.postDelayed (this, 500);
// update last color sent to watch
lastSentColorBackground = pickedColorBackground;
}
};
General impressions about the iron Wear Android
Most of the watches for Android Wear are based on the Snapdragon 400 chip - ASUS, Sony, Samsung and LG use it in all their watches. This chip has an impressive quad-core processor with a frequency of up to 1.2 Hz, which is more than enough for a clock. Its Adreno 305 video card may seem a bit outdated at first glance. But the clock has a rather small resolution of 320x320 pixels, so its power is enough to draw quite complex 3D scenes at 60 frames per second. Even the Moto 360 with its outdated OMAP3630 chip has a fairly powerful PowerVR SGX530 video card, which delivers sufficient performance at the available screen resolution.
For example, the additional effect of the bloom in the resolution of 128x128 with a four-pass blur did not cause any drop in performance - Adreno 305 easily coped with this additional task. However, it was enough for the clock to use a lower resolution of a 64x64 bloom with two blur cycles.
There are a lot of videos where people show games such as Temple Run 2 or GTA running on an Android clock that work without lag - this once again demonstrates the performance of the video cards in these devices.
Shaders used in the application
To reduce the number of drawing commands and the work done by the CPU, the figures are animated in a vertex shader. To illustrate this, here is an example of a transition between the numbers “5” and “0”:
As you can see, the model vertices are divided into 3 groups:
- yellow parts that are not animated - they represent parts that are common between the numbers "5" and "0";
- the dark gray parts will be dropped during the animation - they are not used in the “0” digit;
- the light gray parts will rise during animation; they are used in the number “0” but not in “5”.
This information is entered in the texture coordinates - the V coordinate is set to 0 for fixed parts, 1 for those that go up, and -1 for those that go down. In addition to this, the U-coordinate sets the phase of the animation, so that the cylinders do not move all together, but with a slight lag from each other:
The fragment shader is very simple - the smaller the Z-coordinate of the vertex, the more black the final pixel is drawn. Here’s what the transition between the numbers “3” and “4” looks like (models are mirrored due to RenderMonkey) - you can see how the parts from “4” are shifted down, and the parts “3” are lifted up:
Here is the final result when the bottom of the model merges with the black background:
Vertex Shader Code
uAnim controls the animation, is set in the range [-1..1] uniform mat4 view_proj_matrix;
uniform float uAnim;
uniform float uHeight;
uniform float uHeightColor;
uniform float uHeightOffset;
varying vec2 Texcoord;
varying float vColor;
void main (void)
{
vec2 uv = gl_MultiTexCoord0.xy;
vec4 pos = rm_Vertex;
pos.z + = uHeight * uv.y * clamp (uAnim + uv.x, 0.0, 1.0);
gl_Position = view_proj_matrix * pos;
Texcoord = uv;
vColor = (uHeightColor + pos.z + uHeightOffset) / uHeightColor;
}
uHeight sets the height of the animation - the value depends on the “font” and sets the height of a separate column of the model - equal to 40 units for the font shown in the pictures.
uHeightColor and uHeightOffset allow you to vary the color transition to black. The values uHeightColor = 40 and uHeightOffset = 0 were used for the examples in the pictures, but they are different for different fonts.
Fragment Shader Code
As you can see, almost all the calculations have already been done in the vertex shader - the fragment one only interpolates the colors. uniform vec4 uColor;
uniform vec4 uColorBottom;
varying float vColor;
void main (void)
{
gl_FragColor = mix (uColorBottom, uColor, clamp (vColor, 0.0, 1.0));
}
Link to the archive with shaders for the RenderMonkey program: dl.dropboxusercontent.com/u/20585920/shaders_watchface.rar
Publication
When you publish an Android Wear app on Google Play, you should check the “Distribute your app on Android Wear” box. This initiates the process of moderating the application for compliance with the Wear App Quality requirements. In our case, this process took only a few hours.
Source: https://habr.com/ru/post/255753/
All Articles