Pillow 2.7 - Significant improvement in quality and performance

On January 1, 2015, a new version of the library for working with images Pillow 2.7 was released . Since many changes in it were made by the Uploadcare team, we are pleased to present you an extended version of the release notes for this version.

To begin, remember how it all began. Pillow is a friendly fork (as the authors call it) of the popular PIL library, the Python Imaging Library. The latest version of PIL 1.1.7 was released in 2009 and mainly contained bug fixes. Initially, Pillow was conceived as a project only for putting in order the PIL assemblies, and the developers recommended sending all the bugs not related to the assembly to the original PIL. But as time went on, the PIL rapidly became obsolete, the bugs did not decrease, there still Python 3 loomed on the horizon. Therefore, everything changed with the Pillow 2.0 version. “Pillow 2.0.0 adds support for Python 3 and includes many bug fixes from around the Internet,” reads the project description in PyPI. And since then it started. Every three months there were versions with a huge number of bug fixes and other improvements from various developers. The most significant innovation during this time was, perhaps, support for WebP and JPEG2000 formats. Now it's time for the next big step.

Image resize filters

The image Image.resize() functions Image.resize Image.resize() and Image.thumbnail() as one of the arguments the filter used for resizing - resample . Its possible values ​​are NEAREST , BILINEAR , BICUBIC and ANTIALIAS . The behavior of almost every one of them has changed in the new version.

Image reduction with bilinear and bicubic filters

One of the problems in PIL, and later in Pillow, was that for resizing using a bilinear and bicubic filter, an affine transformation method was used, which uses the same number of pixels of the original image to form one pixel of the final (2x2 pixel for bilinear, 4x4 for bicubic) and fixed filter size. This led to unsatisfactory results to reduce the image, almost no different from the method of the nearest neighbor .
')



On the left is the nearest neighbor method, on the right is the bicubic filter of affine transformations. The first sample is a decrease of 5.8 times, there are practically no differences. The second one is 1.8 times, the differences are minimal, on a sharp diagonal lines a ladder is visible.

At the same time, a high quality convolution based algorithm was used for the ANTIALIAS filter, which gave an equally good result for both reduction and increase.

Starting with Pillow 2.7.0, a high quality convolution based algorithm is used for all three filters.




On the left is a bicubic filter based on affine transformations, on the right - convolutions. Convolutions definitely win.

If before you used any tricks to improve the quality when using a bilinear or bicubic filter (for example, reducing the image in several steps or preliminary blurring), now they are not necessary.

Antialias renamed to Lanczos

A new Image.LANCZOS constant Image.LANCZOS been added to replace Image.ANTIALIAS .

When the ANTIALIAS method was first introduced, it was the only high-quality method based on convolutions. And his name reflected this fact. Now that all methods are based on convolutions, they have all become "smoothing." And the real name of the filter used earlier for this constant is the Lanczos filter.

Of course, the old constant is left for backward compatibility and is a pseudonym for the new one. A joke for linguists: Antialias is alias now.

Lanczos filter quality at magnification

Oddly enough, the quality of the reconciliations was also not all right. In previous versions, there was a bug due to which the quality of the Lanczos filter with an increase was almost the same as that of the BILINEAR filter. This bug has been fixed.




On the left, the result of an increase 4.3 times the previous version, on the right - Pillow 2.7.0. Images on the left are simultaneously more blurred and pixelated.

Bicubic filter quality at magnification

A bicubic filter implemented for affine transformations gave a sharp, slightly pixelated image when zoomed in on. The bicubic filter implemented for convolutions is a bit softer.




On the left, the result of an increase 4.3 times the previous version, on the right - Pillow 2.7.0. The pictures on the left are more pixelated (they have more perceptible pixel boundaries). At the same time, the diagonal lines in the first example are clearer and less susceptible to the effect of a ladder. Both are effects of the parameter “a” in the bicubic equation . Both effects can be avoided only with the help of a better Lanczos filter.

Resize performance

In the general case, convolutions are a more costly algorithm to reduce, because, unlike affine transformations , it takes into account all the pixels of the original image. Because of this, the net performance of the bilinear and bicubic filters may be lower than before. On the other hand, if you were previously satisfied with the quality of the bilinear and bicubic filters to reduce, you may need to consider using the NEAREST filter, which gave almost the same result. This will significantly increase productivity.

At the same time, one of the significant improvements of Pillow 2.7.0 is that the performance of the bundles was reduced on average by a factor of 2 compared with the previous version and even compared with ImageMagick. The performance of the increase in convolutions for the BILINEAR filter was one and a half times faster, for BICUBIC - four times faster, and for LANCZOS remained at the same level.

Since most likely you didn’t use anything other than LANCZOS (formerly ANTIALIAS ) in your application, then the performance with decreasing for you should double on average. If the use of Lanczos was a necessary measure for you because of the poor quality of the remaining filters, then now you can switch, for example, to a bilinear filter. This will increase the performance by about 2 times to reduce and about 30% to increase.

Default filter for Image.thumbnail()

In Pillow 2.5, the default filter for Image.thumbnail() was changed from NEAREST to ANTIALIAS . This filter was chosen for the reason that was repeatedly announced above - the low quality of the remaining filters. In Pillow 2.7.0, the default filter is changed again, this time to BICUBIC , because it is slightly faster. In fact, Lanczos does not offer any advantages after using the Image.draft() method inside Image.thumbnail() , which reduces the image using the libjpeg library and uses Image.thumbnail() for this, not convolution.

Transpose Images

A new method Image.TRANSPOSE been added for the Image.transpose() function in addition to the existing FLIP_LEFT_RIGHT , FLIP_TOP_BOTTOM , ROTATE_90 , ROTATE_180 , ROTATE_270 . TRANSPOSE is algebraic transposition, i.e. reflection of the image relative to its main diagonal.

The performance of the ROTATE_90 , ROTATE_270 and TRANSPOSE methods ROTATE_90 been significantly increased for large images that do not fit in the processor cache.

These three methods are united by the fact that the pixels in them are taken from rows, and placed in columns. Such a memory access pattern is not very effective for large images, because data has time to be pushed out of the processor cache in one pass and they have to be re-loaded from memory for the next pass.

In the new version, the image is divided into logical squares of 128 × 128 pixels in size, and operations on the pixels are performed sequentially within each square. This allows you to significantly reduce the distance that the processor runs on each line, as a result of which the data do not have time to be pushed out of the cache (the memory required for one square is 64Kb).

Gaussian blur and contour sharpness

The implementation of ImageFilter.GaussianBlur been replaced by the consistent use of box filters. The new implementation is based on the article Theoretical foundations of Gaussian convolution by the extended box filtering from the Mathematical Image Analysis Group. Since the implementation of ImageFilter.UnsharpMask is based on Gaussian blurring, everything described in this section also applies to it.

Blur radius

In previous versions of Pillow, there was an error due to which the blur radius (standard deviation of Gaussians) actually set its diameter. Therefore, for example, to blur an image by radius 5, it was necessary to specify the value 10. The error was corrected, and now the radius value is interpreted in the same way as in the rest of the software.

If you previously used a Gaussian blur with a certain radius, you need to divide its value by two.

Blur performance

The calculation time of the box filter is constant relative to its radius and depends only on the size of the input image. Since The new implementation of Gaussian blur is based on the box filter, its calculation also does not depend on the blur radius.

For a radius of 1 pixel, the new implementation runs 5 times faster, for a radius of 10 - 18 times, for a radius of 50 - already 85 times. Your designer who draws iOS 8 interfaces should be pleased.

Blur quality

Theoretically, with Gaussian blurring, all points of the original with certain coefficients should be involved in the calculation of each point of the final image. In practice, the coefficients of points beyond 3 × standard deviation are so small that it makes no sense to take them into account.

The previous implementation took into account only pixels within a radius of 2 × standard deviation for each final pixel. This was not enough, so the quality was worse compared to other implementations of Gaussian blur.

Despite the fact that the new implementation is only a mathematical approximation, it does not contain such a bug.




On the left, the result of a blur with a radius of 5 in the previous version (including a bug with a doubling of the radius), on the right - in a new one. On the left, sharp boundaries of objects are visible.

---

All these changes are already working on our servers. Thanks to them, we have improved the quality and speed of the API for processing images on the fly . We also implemented a quick blur operation. But that is not all. We are preparing the next big step for Pillow, which we will announce later.