Although Discord is an application for voice and text chat, over 100 million images pass through it every day. Of course, we would like the task to be simple: just redirect the pictures to your friends across all channels. But in reality, the delivery of these images creates quite large technical problems. A direct link to the pictures will give the host with a picture the IP addresses of the users, and the big images will consume a lot of traffic. To avoid these problems, an intermediate service is required, which will receive images for users and resize them to save traffic.

Meet the Image Proxy

To do this, we created a Python service and creatively named it Image Proxy . It downloads images from remote URLs, and then performs the resource-intensive task of resizing using the pillow-simd package . This package works surprisingly fast, using where possible to speed up the resizing of the x86 SSE instructions . The Image Proxy will receive an HTTP request containing the URL to download, resize, and finally give the final image.

On top of it, we set a caching layer that saves images with resized in memory and tries, if possible, to output it directly from memory. The HAProxy layer redirects requests based on the URL hash to the Nginx caching layer. The cache merges queries to minimize the number of transformations required by resizing images. The combination of cache and proxy allowed us to scale our resizing service to millions of users.
')


As Discord grows, the Image Proxy service has begun to show signs of overload. The biggest problem was that the load was distributed unevenly, because of which the bandwidth suffered. Requests were executed at completely different times, up to a few seconds. We could solve this problem in the existing Image Proxy, but at that time we were experimenting with the additional use of Go, and here it seemed like a great place to use Go.

So appeared Media Proxy

Rewriting a service that is already working has become a difficult decision . Fortunately, the Image Proxy is relatively simple and you can easily compare the results from it and the new alternative. In addition to faster query execution, the new service has some new features, including the ability to extract the first frames of the .mp4 and .webm video clips — hence, let's call it Media Proxy.

We started by measuring the performance of existing image resizing packages on Go and quickly became disheartened. Although Go is generally a faster language than Python, none of the packages found exceeded reliably in speed Pillow-simd. The main part of Image Proxy's work was transcoding and resizing images, so it would definitely become a bottleneck in the performance of Media Proxy. Go can be a little faster when processing HTTP, but if it is not able to quickly resize images, then the additional speed gain is leveled by additional time for resizing.

We decided to double the bid and build our own library of image resizing on Go. Some promising results were shown by one Go package based on OpenCV , but it did not support all the functions we needed. We created our Go resizer called Lilliput with our own Cgo wrapper over OpenCV. When creating, we watched carefully so as not to generate excess garbage in Go. Wrapper Lilliput does almost everything we need, although it took a little fork OpenCV for our needs. In particular, we wanted to be able to check the headers before deciding on decompression of images: this way you can instantly refuse to resize too large images.

Lilliput uses existing and tested C libraries for compression and decompression (for example, libjpeg-turbo for JPEG, libpng for PNG) and vectorized OpenCV code for quick resizing. We added fasthttp to meet our parallel HTTP client and server requirements. As a result, such a combination allowed launching a service that invariably exceeded the Image Proxy in synthetic benchmarks. In addition, lilliput worked no worse or better than the pillow-simd in those tasks that we need.



The first code was not without problems. Initially, the Media Proxy issued a leak of 16 bytes for each request. This is small enough to be noticed immediately, especially when testing on a small volume of queries. To solve the problem, large static pixel buffers were installed in Media Proxy for the purpose of resizing. It uses two such buffers on the CPU, so that the 32-core machine immediately takes 32 gigabytes of memory. During testing, restarting the Media Proxy took several hours, since it uses absolutely all of the memory. This is quite a long time to make it difficult to clarify the situation: whether we really have a memory leak, or just exceeding the limit during operation.

In the end, we decided that there must be some sort of memory leak. We weren't sure if this was a leak in Go or C ++, and studying the code did not give an answer. Fortunately, Xcode comes with an excellent memory profiler - the Leaks tool in the Instruments menu . This tool showed the size of the leak and the approximate location where it occurs. Such a hint was enough for a more thorough study to determine the source and fix the leak.

In Media Proxy, we are faced with another bug, which had to be interrupted. Sometimes he gave out strangely spoiled pictures, where one half remained normal, and the other was "buggy." We suspected that we were partially encoding the image somewhere or somehow incorrectly calling OpenCV. The bug was infrequent and difficult to diagnose.

To advance the investigation, we developed a high-performance query simulator that issued URLs with a link to the HTTP server in the simulator, so this simulator worked simultaneously as a requesting client and host server. He randomly inserted delays in responses to trigger such image corruption in the Media Proxy. By reliably reproducing the problem, we managed to isolate the components of the Media Proxy and find the status of the race in the output buffer containing the resized image. One image was written to this buffer and then another before the first one was returned to the system. The glitch pictures in reality were two JPEGs, recorded one on top of the other.


Real buggy jpeg generated by media proxy

Another way to look for bugs in complex systems is fuzzing, when random input data is generated and sent to the system. In this case, the system may show strange behavior or collapse. Since our system must be resistant to any input data, we decided to use this important technique in the testing process. AFL is an exceptionally good fuzzer , so we chose it and set it on Lilliput, which allowed us to identify several failures due to uninitialized variables.

After correcting these bugs, our confidence grew enough to roll out a media proxy into production - and we were glad to see that our efforts were worth it. Media Proxy required 60% less server instances to process the same number of requests as the Image Proxy, performing these requests in much smaller time spreads. Profiling showed that more than 90% of the time of the CPU in the new service is spent on decompression, resizing and compression. These libraries have already been significantly optimized, that is, additional growth would not be easy. In addition, the service almost did not generate garbage in the process.

Now Media Proxy performs image resizing with a median time of 25 ms, and the response delay is 85 ms over the median. It changes the size of more than 150 million images every day. Media Proxy runs on an auto-scalable GCE host group of type n1-standard-16 , with a peak number of 12 instances on a typical day.

Media download in Media Proxy

After successful service of the service on static images, we wanted to connect support for resizing animated GIFs to it, and OpenCV will not do this work for us. We decided to add another Cgo wrapper to Lilliput over giflib so that Lilliput could resize the animated GIFs as well as output the first frame in PNG format.

Changing the size of the GIF was not so easy, because the GIF standard provides for using palettes of 256 colors in each frame, and the resizing module works in RGB space . We decided to save the palette of each frame, rather than calculate the new palette. To convert RGB back to palette indices, we supplied Lilliput with a simple reference table that took some of the RGB bits and used the result as a key in the palette index table. It worked well and retained the original colors, although this approach does not mean that Lilliput is able to create GIFs only from source files of this format.

We also patched giflib to make it easier to decode just one frame at a time. This allowed the frame to be decoded, resized, then encoded and compressed before moving on to the next. This reduces the memory consumption of the GIF resizing module. This complicates Lilliput somewhat because it needs to save some GIF states from frame to frame, but more predictable memory usage in Media Proxy looks like a clear advantage.

The giflib wrapper in Lilliput fixes some of the problems that were present in the Image Proxy GIF-Resizer, since giflib gives you complete control over the resizing process. A considerable number of Nitro users download GIF-animated avatars that are buggy or have transparency errors after resizing in Image Proxy, but work fine after processing Media Proxy. In general, as it turned out, resizing programs have problems with some aspects of the GIF format, so they give out visual glitches for frames with transparency or partial frames. Creating your own wrapper allowed us to solve the problems we encountered.

Finally, in Lilliput they added a Cgo wrapper on libavcodec , so that it could stop the video and get the first frame of the MP4 and WEBM clips. This functionality will allow the Media Proxy to generate thumbnails for videos published by users, so that other people, based on this thumbnail, make a decision whether to run the video or not. The extraction of the first frame remained one of the last factors that stopped us from adding a built-in video player to the client for published files and links.

More Open Source

Now that we are satisfied with the work of the Media Proxy, we publish Lilliput under the MIT license. Hopefully, the package will be useful to those who need a productive service for resizing images, and this article will encourage others to create new Go packages.