Adding ColorKey to libGDX

Hi Habr! In this article, I will talk about adding a color key to the libgdx library (or any other where there are shaders). As you know, libgdx doesn’t have native support for “transparent color”, so you have to store a full-color image in RGBA8888 format (4 bytes per pixel) or in a truncated RGBA4444 format (2 bytes per pixel), which reduces memory usage by half, but makes it worse picture When creating 2D games, often it would be enough just one bit of transparency ... Especially on mobile platforms ... but it is not ... Let's make it happen!

RGBA8888

First we need a reference image in the RGBA8888 format, with which all subsequent attempts to save bytes will be compared. We will experiment with a set of tiles from which a level is drawn, we don’t pay attention to the sky and the little man. The tile size is 512 a 512 pixels. On the disk, the textures are saved in 8-bit png with transparency and occupy 20 kilobytes (you can save to 32-bit png, with the same result, because the graphics are simple, and there is always transparency, or it is not). In video memory, they will occupy 512 * 512 * 4 = 1 megabyte exactly. Hotstsa smaller, it's not the only texture ...

RGBA4444

First of all, the idea arises to use a truncated bit. Pixel art is simple, there are few colors, and we will immediately save 512 kilobytes. We try:


')
The grass only slightly changed the shade, you can put up with it, but here the stones suffered critically. If you are ready to accept it, then you can not read further. I'm not ready.

We write a shader!

Without further ado, I copied the shaders by default and modified the fragment shader:

private static String fragmentShader = "#ifdef GL_ES\n" // + "#define LOWP lowp\n" // + "precision mediump float;\n" // + "#else\n" // + "#define LOWP \n" // + "#endif\n" // + "varying LOWP vec4 v_color;\n" // + "varying vec2 v_texCoords;\n" // + "uniform sampler2D u_texture;\n" // + "void main()\n"// + "{\n" // + " LOWP vec4 pixel = texture2D(u_texture, v_texCoords);\n"// + " gl_FragColor = v_color * pixel;\n" // + " if( pixel.rgb == vec3(1.0, 0.0, 1.0) ){gl_FragColor.a = 0.0;}\n"// + "}"; 

Only the last three lines are interesting. First, save the texel color (Important! Texture interpolation must be disabled, that is, we use NEAREST filtering when loading). Then we set the pixel color by multiplying the texel color by the vertex color. If you don't add vertex colors, then this multiplication can be replaced by assignment. And finally, we compare the color of the texel with the “transparent color” and, if the colors are the same, then we make the pixel transparent. As a “transparent”, I chose the classic vyglazno-purple rgb (255,0,255). Surely you can get rid of the conditional operator, but ... "And so it will come down!".)

RGB565

Now we don’t need to spend 4 bits to store 1 bit of transparency and we can spend more bits on storing color information. This is what came out of this: the vyvlaglazik became transparent, and the loss of information about the color to the eye is not distinguishable (depending on the input image, it can become quite distinguishable, especially on gradients).

That's how we easily and naturally reduced memory consumption by half, almost without loss of quality and speed (after all, I want to get rid of the conditional operator in the shader). But I want more. I want to compress the texture in ETC1 format, but with transparency. Still, six times less than the RGB. and not on disk, but in memory!

We try ... Epic file. Expected. The title picture is the result of this attempt. The result is expected, because ETC1 is a lossy compression format. With heavy losses. Vigor-eye color dimmed and pixels of half-eye-green color appeared. Usually, the alpha channel is stored in a separate texture. Often - without compression. But this is not our method! Let's see what can be achieved if you fool around a bit with the shader.

Shader for entertainers

  if( vec3( min(pixel.r,0.95), max(pixel.g,0.05), min(pixel.b,0.95)) == vec3(0.95, 0.05, 0.95) ) { gl_FragColor.a = 0.0; } 

Replace only the last line in our shader. Now we are not comparing strictly with a specific color, but with a slight deviation: we allow the red and blue components to be slightly darker, and the green component to be lighter.

There is no need to even compare with the original, artifacts are visible with the unaided eye. But! If you play with a tolerance, or read the “distance” between colors (also with a sufficient deviation), then it is quite possible to achieve tolerable results for a particular set of textures. When you fight for every kilobyte - this method may be quite acceptable.

Transparent jpeg?

Why not? We already have a neck that will make any pasture transparent. If you're lucky, the result will even be usable. If the disk space is important, and png is too bad, then why not. Let's try two options at once: with the compression profile "maximum" and "very high"

We see that with the profile “maximum” it is quite possible to use jpg with “transparent color”. In theory. If using png is less profitable.

So, we managed to halve the occupied memory, almost without losing the color of texur, but getting a "transparent color" to indicate completely transparent areas. As a bonus, they learned to make transparent jpg.

I hope the note will be useful not only for me. Even more I hope that someone will offer an equivalent code without a conditional operator. Thanks for attention.

UPD :

User FadeToBlack offered two shader options without a conditional operator:

This shader can only be used with textures in which transparency is indicated through a color key. Textures with real transparency will not display correctly. The shader from the article correctly handles both textures with real transparency and “transparent color”.

  void main() { LOWP vec4 pixel = texture2D(u_texture, v_texCoords); gl_FragColor = v_color * pixel; gl_FragColor.a = float(pixel.rgb != vec3(1.0,0.0,1.0)); } 

And this shader is equivalent to the shader in the article, but without a conditional operator. The translucency of the entire sprite can be set through the color of the vertices of the sprite, regardless of the transparency of the texture.

  void main() { LOWP vec4 pixel = texture2D(u_texture, v_texCoords); gl_FragColor = v_color * pixel; gl_FragColor.a = gl_FragColor.a * float(pixel.rgb != vec3(1.0,0.0,1.0)); }