Porting Android applications to WebGL

The idea of ​​porting an Android application to WebGL

Now WebGL is supported by almost any device and works quite stably and quickly even on mobile devices, so it was very interesting to try to implement something on this technology. We already have a lot of experience with OpenGL ES 2.0 in Android - we have created quite a few different three-dimensional live wallpapers.


We created these applications without using ready-made third-party engines (for example, Unity) or high-level frameworks for working with OpenGL (such as libGDX). Due to the fact that the code is not burdened by the limitations of third-party frameworks, the applications are very small and fast, and we also have the ability to fully optimize the rendering for the needs of each application.

WebGL is based on OpenGL ES 2.0, and therefore the process of porting wallpaper is quite simple and straightforward.
')

Rendering framework

As well as in the original Java code, the base classes BaseRenderer and BaseShader are implemented . It was decided to use ECMAScript 2015 classes since this notation simplifies the readability of the code, and the only browser that does not support the JS classes is IE11. BaseRenderer contains code for creating a WebGL context, initializing, tracking window resizing, etc. It also has empty stubs for rendering the scene. BaseShader contains code for compiling and using shaders. The rendering of the scene itself and the loading of data for it are implemented in BitcoinRenderer .

Downloading ready data

Most WebGL engines and demos load data from files in JSON, OBJ, or other formats. On the one hand, this is convenient - just export models from Blender or 3ds Max and use them in the scene. However, on the other hand, this approach requires additional processing of source data on the client to create buffers with data ready for use by the video card. Also, the data in these formats often contain a lot of redundant information, which, although not used, still takes up most of the original file, which significantly increases the amount of data transferred. Together, these two drawbacks lead to the fact that often even unpretentious WebGL demos are loaded and run for quite some time.

In the Java version of our framework, we use binary data ready for direct loading into OpenGL buffers, and in the JS version we use the same approach. XMLHttpRequest Level 2 supports working with binary data in JavaScript. To simplify working with XHR2, a simple BinaryDataLoader class has been created .

The FullModel class provides work with meshes. The load () method loads two buffers for the model — with indexes and data (vertex coordinates, UV coordinates, etc.). These buffers contain binary data ready for use by the video card. The class also has a bindBuffers () method that actually binds the buffers and must be called immediately before glDrawElements () .

Compressed textures in ETC1 format

To save video memory in live wallpapers we use various compressed textures. Our Java framework supports the ETC1, ETC2, PVRTC and ASTC formats and uses the most appropriate textures based on the capabilities of a particular device. WebGL implements only ETC1 and uncompressed RGB textures.

In OpenGL ES 2.0, ETC1 is a mandatory part of the standard and is supported on all devices without exception. However, in WebGL, support for ETC1 compression is not necessary, and the presence of the WEBGL_compressed_texture_etc1 extension should be checked. All desktop browsers except IE11 and Edge support this extension. Microsoft browsers have to use uncompressed textures.

You can check which texture formats are supported by the browser using this handy page .

Through the use of ETC1 textures, we use much less memory and also managed to speed up the process of loading textures. After all, uncompressed textures must first be decoded from the original format (PNG, JPEG, WebP, GIF, etc.) to bitmap (RGBA or RGB, with or without alpha channels) before they get into the video memory, and then transferred to the driver for loading into the video card. ETC1 textures do not require any preprocessing - they are already ready for direct use by the video card and therefore load much faster.

If we talk about saving memory, for example, an uncompressed RGB texture with a size of 512x512 pixels takes 768 kb, while the same ETC1 texture takes only 128 kb. However, ETC1 is not perfect and causes some compression artifacts. These artifacts are almost invisible on diffuse maps and lighting maps, but very noticeable on normal maps (distortions in the form of 4x4 pixel blocks) and reflection maps (inaccurate color rendering). So we use both compressed and uncompressed textures depending on quality requirements.

In Android, loading ETC1 textures is very simple - there is a standard utility ETC1Util , which does all the work of loading textures from a file in the PKM format. Due to the fact that WebGL does not provide any means for loading compressed textures from known formats, I had to create my own ETC1 texture loader from files in PKM format. PKM is a very simple format, it consists of a header of 16 bytes, followed by binary data, ready to be loaded into a video card. More information about the title can be found here and here . When writing the code for getting the texture size, we faced a certain limitation of JavaScript. These values ​​are stored as 16-bit big-endian integers. However, using Int16Array to get these numbers will not work, because JavaScript does not provide a way to set the byte order of the buffer, and I had to read bytes using Uint8Array and calculate 16-bit values ​​manually from the resulting low and high byte pairs.

Shaders

The application uses only two shaders: one for the surface of the table, and the other for the model of coins.

The shader for the table is a simple implementation of lightmaps. It uses two textures: one color map (diffuse) to set the texture of the tree, and the second - lightmap (lightmap). The result of the shader is simply the multiplication of colors from these textures.

The coin shader is more complex; it contains the following functions:

• Spherical reflection map (so-called sphere mapping or matcap)
• Lightmap with color enhancement
• Normal map

A spherical reflection map stores information in a picture that looks like a snapshot of a chrome ball:



Compared to other reflection techniques, the spherical map has the following advantages:

+ Simple shader code and high performance
+ Use only one texture
+ Texture with reflection easy to process in graphic editors, for example in Photoshop
- Poor performance on flat surfaces.
- Part of the texture space is wasted (corners are not used)

The greatest advantage of a spherical map is ease of implementation. When you already have a normal calculated in screen space, you just need to take its (x; y) component to read the texture. Extract from the shader code:

vec4 sphereColor = texture2D(sphereMap, vec2(vNormal2.x, vNormal2.y)); 

This also implies the greatest drawback of this technique: since only the screen normal is involved in the calculations, the large smooth polygons will be filled with the same color (since they will have the same normal over the entire surface). And the coin model consists mainly of large flat surfaces. To combat this drawback, we added a normal map and made it as diverse as possible: added noise, labels, numbers, etc. This technique deprives the model of flat surfaces and spherical reflections work perfectly.

When lighting the coins was used an extra trick. For shading coins, a lightmap is used. However, a simple multiplication of color on the irradiance map gives a more or less correct, but boring result: darkened places become just darker. In addition to this, in dark places, we multiply the color by ourselves using the pow () function. The degree is the higher, the darker the irradiance map. This reproduces the effect that light enters the “trap” in an enclosed space and enhances its color due to multiple reflections from metal surfaces. The result is a more realistic metal surface:

Result

The finished demo can be viewed on this page . All sources are available on GitHub and you can use them in your projects under the MIT license.