Insane GIF Converter in Animated Telegram Stickers

Instead of a thousand words...


xZibit is also happy, because here GIF is inserted into stickers to be inserted into GIF for KDPV!

And now about implementation details.

It all started with a discussion in the Telegram-developers chat about the upcoming feature:
')


The man said - man did! The first prototype on Pillow and svgwrite, which parses GIFs into pixels and converts them into vector squares with a preview in SVG, was written in one output.

The fun started on ...

JSON is an open format, they said ...

Until now with the formats in the Telegram now and then tricky. Made support for GIF-animations - in fact, they are converted to MP4-video. Made support stickers - they are unloaded in PNG, but converted to WebP. This time everything is more honest: what is at the entrance, then what is at the exit.

For animated stickers in Telegram, GIF is not used, not video, and not even any established vector graphics format like SVG or - let Cthulhu! - Flash. It involved a new-fashioned format that came out from under the wing of the Airbnb - Lottie. Hitherto, he had some fame among mobile developers, but thanks to Telegram, he may be more popular.

In essence, Lottie files are Adobe After Effects projects serialized in JSON, which implement all the features of this program to the maximum. With the display, alas, everything is not so rosy . Although there are a lot of ready-made "official" implementations of the library for rendering Lottie, just under the Telegram-covered platforms: Android, iOS, Qt and Web - only a part of the format capabilities is implemented in all of them. Telegram went even further and limited the list of supported features, and also “invented” their own format, which differs from the usual Lottie only in GZip packaging and in the "tgs": 1 parameter "tgs": 1 . I think I know where Denis Popov is working now! :-)

And if with documentation on libraries for different platforms, everything is pretty good, then, alas, it was not possible to find at least some description of the format device - only the JSON scheme in the lottie-web source code. It was necessary to pick the existing animations along the way in order to understand the general concepts of the format. There were also discrepancies between real files and the layout: in particular, in layers of type 4 , according to the diagram, nested objects are stored in the "it" property — however, in real files the key is called "shapes" , and "it" does not work.

Clarified format nuances:

• The file consists of layers. Unlike GIF, here each layer can have an arbitrary start and end time for the display. You can (more precisely, need ) apply various transformations to the layer: scaling, rotation, changing transparency, etc. Layers can even be three-dimensional (prohibited for Telegram).
• A layer consists of shapes. They have many types, some cannot be used in Telegram. In practice, in order for a layer to appear, it must include three shapes: a contour (in finished animations it is usually the type "sh" - Bezier curves; the converter so far uses only the type "rc" - rectangles), the fill (type "fl" ) and the transformation ( type "tr" ).
• You can even include raster elements, create text layers, set relationships of layer parameters and shapes through expressions. All this goodies are also banned in Telegram.

The first problem directly follows from this: redundancy . Although the JSON schema has recently added default values ​​for transformation parameters - they are not implemented in libraries. So it’s still necessary to set them explicitly.

It would seem that this is not a problem at all? Even a simple GZip does a good job with compressing cryingly repetitive data, and 1 MB of raw JSON magically transforms into a couple of tens of kilobytes, which quietly creep into the stated limit of 64 KB. Not here it was!

I upload, it means, a chubby animation that calmly displays a lottie-web, in a Telegram - and here, instead of the relatively beautiful pixel art, the static blurred look at me here:



What?! But it turned out that there is also a clearly not indicated 1 MB limit for the decompressed data. A representative of the Telegram team promptly confirmed it and announced the imminent increase of the limit to 2 MB.

Even if these problems are resolved - stickers that go beyond 1 MB of uncompressed data and do not contain transformations will not be available to users of older versions of Telegram. So it is necessary, apparently, to observe the restrictions in the future.

Transparency is important

Pillow, along with OpenCV, can be called the industry standard for image processing in Python. Moreover, it is not bad sharpened and under the features of GIF: it supports indexed colors, gives access to the palette. It supports the conversion of a pixel map to a NumPy array, which is important for productive processing. Even collects color statistics! But there were also minuses:

1. There was no documented way to get a transparent color index. It was necessary to imply, as a temporary solution, that the transparent color is the most common, but in real GIFs this is not always the case.
2. The same with the delay between frames: Pillow gives only the frames themselves as a sequence of images, without information about delays.
3. Sometimes partial frames are incorrectly overlaid.

Therefore I had to look for a replacement. The gif2numpy module was used as it. It is “sharpened” for GIF features and provides access to all the technical properties of both the image and individual frames, including GCE . Thus, it solves the problem of reading delays.

Transparency, as it turned out, doesn’t support gif2numpy at all: colors are immediately converted into three channels with a bit depth into bytes, without taking into account the bit depth and saving color indices. Fortunately, the module consists of a single file, so it was not difficult to include it in the project and modify it, reserving the color #FE00FE for transparency.

The problem with partial frames was not trivial to solve. gif2numpy tries to impose such frames on the previous one, but does not check the blending parameters, which is why the correct result does not always work either. In order not to bother with the flags, added pre-processing of images using gifsicle with the key - --unoptimize - it converts partial frames to full. And at the same time it leads them to use a global palette, which eliminated the need to separately process a transparent color when using our own frame palette.

Squeeze me harder

Squares are good, but with such restrictions you need to show more imagination, otherwise even miniature GIFs do not “crawl through” in Telegram.

The first thing to go was something similar to RLE : horizontal squares of the same color combined into one rectangle.

Next is the turn of operating Lottie features. Since each layer has an arbitrary start and end time, you can apply the technique that video codecs used a long time ago, and partly in the GIF itself: the squares that remain in one place for several frames can be merged into one layer, which is replaced during display. several others. What is implemented, so far only for pairs of adjacent layers.

Development plans

Ideas that can be applied here in bulk:

• Recognize monochrome areas of any size. You can break them into a set of rectangles, for which there is a good algorithm . It is also advisable to convert them to a contour, but this is overshadowed by the need to indicate all points of the Bezier curves in the Lottie — rectangles in some cases may be more advantageous.
• Recognize movement. Technology, again, since ancient times used in video codecs. If the same contour does not change shape from frame to frame, but only coordinates - instead of duplication on several layers, it is necessary to place it on one layer with transformation.
• Recognize the "covering" of some areas by others. Example:
  
  

   ...... .O..O. ...... .OOOO. ...... 

  
  On a rectangle of one color superimposed pixels of another color. Instead of dividing this rectangle into a pile of small ones, or transposing them into a contour of complex shape, you can simply lay them on top of the whole rectangle.
• Vectorization of curves and ellipses, gradient recognition. This will spoil the pixel charm, but on the orders will improve the compressibility of some GIF'ok. There are gradients even in antediluvian "koloboks", I guarantee it!
• Compression lossy. In the first place - the elimination of dithering, and in the unnecessarily smoothed images does not hurt the color restrain. The aforementioned gifsicle will certainly cope with this.

Links

• Sources Scary places.
• The channel on which I post packs successfully converted GIF'ok.