Compressing the incompressible - how to reduce the distribution of a mobile application
Today, in app stores for iOS and Android platforms, there is an application size limit of 100 MB. Apple Store for applications that do not fit into this limit, prohibits download using the mobile Internet. On Google Play, this is a strict limit on the size of the APK - everything that does not fit into it should be placed in the add- on files. For users with paid traffic, downloading a large application can be quite expensive, so you should try to reduce its size with all your might.
As part of this article, we will describe with what tricks we were able to meet this restriction on the Gardenscapes project for the iOS platform. The article concerns mostly mobile games, but compression methods are universal and can be useful for any projects with heavy graphics. In order to talk about compression methods, you need to decide how the archive of the application is formed.
Applications on the iOS platform are distributed as binary files with the .ipa extension. These compressed .zip-folders, which include the executable file of the application (for the processor with the ARM architecture), as well as application resources. Using the .zip format means that your ipa file can be opened using any convenient utility for working with archives and examined. More specifically, Apple uses its own LZFSE format, and it’s not so easy to unpack the archive. However, the contents of the archive can be read without additional effort. For example, let's look at the contents of ipa in the Gardenscapes project:
Resource Allocation Chart in .ipa file
Most of the resources are occupied by the game, while the executable file is almost not inferior to them. The system resources and the SDK resources at the same time occupy a fairly small amount of space in the archive, so it makes no sense to optimize them more often.
')
The executable file takes up so much space for two reasons:
Starting with iOS 9, the first item is no longer relevant, since Apple is using App thinning technology, collecting different packages for different architectures. From a developer’s point of view, this works transparently: you send one application to submit as before. And the user downloads only a 32-bit or 64-bit executable file, depending on the device. Therefore, if you can afford not to support devices with iOS versions below 9, then the size of the executable file can be divided roughly in half. At the same time, users with versions of iOS 8 and below will download the archive containing both architectures.
Point 2 is always relevant. When calculating the estimated IPA size, the uncompressed size of the executable file must be considered. At best, it will shrink only slightly. That is why Apple does not recommend storing large amounts of data (images, large text resources, etc.) in an executable file, this can lead to an abnormally large application size. You should also check the optimization settings in Xcode and make sure that you do not connect extra libraries.
For mobile games, you can select groups of resources, according to the list from heavier to less heavy:
Trying to compress the last two groups of resources is not very appropriate, because ipa is a zip archive. Text data is usually compressed sufficiently without additional effort.
The size of the graphics can be reduced in two ways:
Our projects mainly use png, webp and pvrtc:
Do not forget that compressed formats have their own quality settings, and they can also be customized.
Compression example
An example of the same webp image compressed with a quality setting of 85, 95 and 100. The difference is noticeable, but even the most compressed image looks acceptable. At the same time, the difference in size between webp85 and webp95 is about 3 KB.
Webp needs to be used carefully due to unpacking speed issues. On slow devices, loading such an image can produce noticeable lags. You should not use it for packing large graphics, which is loaded on the fly.
The table shows the sizes of the atlas 2048x2048 under compression in different formats:
As you can see, the png format should not be used at all. From webp and pvrtc formats, you can choose the appropriate one, taking into account the requirements for the quality of graphics and the speed of its loading.
Gardenscapes mainly use animations with keyframes that are exported from .swf. To save space in such animations, you should try to optimize the number of frames per second (FPS). In our draft standard, a frequency of 24 FPS is chosen, the animations do not lose in quality and are rather compact. In addition to the frame rate, the amount of detail affects the size of the animation. If the animation contains a lot of details, its size will also be large - because for each detail will be stored its own set of key frames. For additional compression, we use a special lossy-codec for animations. We also use FSE encoding, more information can be found here .
There are no general recommendations here; in general, it all comes down to fine-tuning the encoding formats. We cannot get any phenomenal winnings here, we managed to compress the video by about 30% compared to the original version and save about 10-20% on sounds. It all depends on the specific video and audio - you need to select the settings experimentally.
Typically, fonts do not take up much space in most applications. But this is until you come across language localizations with hieroglyphic writing. The number of hieroglyphs there can be measured in thousands. This increases the font files up to several megabytes. To combat this, we use an automated script that lists the characters based on the game text file — all unused characters are removed from all fonts.
This is usually the smallest group of resources, and it does not need to be separately optimized. The ipa file itself is an archive, and the text data is well archived. In our project, however, the texts take up quite a lot of space. Mainly due to the need to support many localizations.
The size of the build is really a problem for us. How serious it is, you can understand by some additional methods of compression that we have invented.
Build size optimization is an extensive topic, this article describes only general principles and methods of resource compression. In addition, there are usually other difficult questions:
The techniques listed in the article are the experience of our development team in a particular project. But they are quite versatile, and they can be used not only for mobile games, but also for other applications.
While the article was being prepared for publication, Apple increased the maximum build to 150 MB. It would seem to be better. But the Google Play limit is still 100 MB. As a result, everything turns out even harder. If we are in the preparation of the build for iOS, we will focus only on the restriction of the Apple store to 150 MB, then after preparing the build for Android we just can’t meet the 100.
You need to make different builds for iOS and Android in order to make the best use of the capabilities of the app stores. Now we are still in the process of finding the best solution. And we are looking forward to a similar step to increase the limit from Google. After that, you can simply replace the number 100 by 150 in the text of the article, and everything will be correct again in it.
As part of this article, we will describe with what tricks we were able to meet this restriction on the Gardenscapes project for the iOS platform. The article concerns mostly mobile games, but compression methods are universal and can be useful for any projects with heavy graphics. In order to talk about compression methods, you need to decide how the archive of the application is formed.
Apple App Store Format
Applications on the iOS platform are distributed as binary files with the .ipa extension. These compressed .zip-folders, which include the executable file of the application (for the processor with the ARM architecture), as well as application resources. Using the .zip format means that your ipa file can be opened using any convenient utility for working with archives and examined. More specifically, Apple uses its own LZFSE format, and it’s not so easy to unpack the archive. However, the contents of the archive can be read without additional effort. For example, let's look at the contents of ipa in the Gardenscapes project:
Resource Allocation Chart in .ipa file
Most of the resources are occupied by the game, while the executable file is almost not inferior to them. The system resources and the SDK resources at the same time occupy a fairly small amount of space in the archive, so it makes no sense to optimize them more often.
')
Optimization of the size of the executable file
The executable file takes up so much space for two reasons:
- It contains the binary code for two architectures (32 and 64 bits);
- The executable file in ipa is encrypted, which greatly impairs compression.
Starting with iOS 9, the first item is no longer relevant, since Apple is using App thinning technology, collecting different packages for different architectures. From a developer’s point of view, this works transparently: you send one application to submit as before. And the user downloads only a 32-bit or 64-bit executable file, depending on the device. Therefore, if you can afford not to support devices with iOS versions below 9, then the size of the executable file can be divided roughly in half. At the same time, users with versions of iOS 8 and below will download the archive containing both architectures.
Point 2 is always relevant. When calculating the estimated IPA size, the uncompressed size of the executable file must be considered. At best, it will shrink only slightly. That is why Apple does not recommend storing large amounts of data (images, large text resources, etc.) in an executable file, this can lead to an abnormally large application size. You should also check the optimization settings in Xcode and make sure that you do not connect extra libraries.
Resource Optimization
For mobile games, you can select groups of resources, according to the list from heavier to less heavy:
- Game graphics;
- Animations;
- Video and sounds;
- Fonts;
- Text resources and text configuration files.
Trying to compress the last two groups of resources is not very appropriate, because ipa is a zip archive. Text data is usually compressed sufficiently without additional effort.
Compression game graphics
The size of the graphics can be reduced in two ways:
- Closely pack graphics in polygonal atlases to save at the expense of transparent pixels. Polygonal atlases are more intended to save RAM, but the size of files with atlases also becomes somewhat smaller.
- Divide the graphics into different groups according to the required quality and pack each group into its own compressed format.
Our projects mainly use png, webp and pvrtc:
- png: lossless compression format. It has the best quality, but it takes up a lot of disk space. This format is not recommended for use in the application, since the lossless version of the webp is more optimal in size. However, the boot screen and application icons in ipa must be in .png format. To optimize the size of .png, you can use the optipng utility available at Mac Ports.
- webp: lossy-option c acceptable size. Allows you to change the quality for different types of images, but unpacked long enough. There is also a lossless option.
- pvrtc: a hardware-supported compressed format for iOS. It loads quickly, takes up very little space. However, there are a number of limitations. First, the image must necessarily be a square with a side equal to two.
Secondly, pvrtc is a lossy format, and even very lossy. It should be used carefully, do not use with graphics that require high quality. Bad for graphics with transparent areas, there will be artifacts on the border. The main benefit of the pvrtc format is the savings in RAM. Unlike the png and webp formats, it does not need to be unpacked into RGBA for drawing.
Do not forget that compressed formats have their own quality settings, and they can also be customized.
| Webp 100 | Webp 95 | Webp 85 |
 |  |  |
An example of the same webp image compressed with a quality setting of 85, 95 and 100. The difference is noticeable, but even the most compressed image looks acceptable. At the same time, the difference in size between webp85 and webp95 is about 3 KB.
Webp needs to be used carefully due to unpacking speed issues. On slow devices, loading such an image can produce noticeable lags. You should not use it for packing large graphics, which is loaded on the fly.
The table shows the sizes of the atlas 2048x2048 under compression in different formats:
| Format | png | pvrtc | webp lossless | webp 95 quality | webp 85 quality |
| Size, KB | 8057 | 2049 | 5593 | 2218 | 1579 |
As you can see, the png format should not be used at all. From webp and pvrtc formats, you can choose the appropriate one, taking into account the requirements for the quality of graphics and the speed of its loading.
Animations
Gardenscapes mainly use animations with keyframes that are exported from .swf. To save space in such animations, you should try to optimize the number of frames per second (FPS). In our draft standard, a frequency of 24 FPS is chosen, the animations do not lose in quality and are rather compact. In addition to the frame rate, the amount of detail affects the size of the animation. If the animation contains a lot of details, its size will also be large - because for each detail will be stored its own set of key frames. For additional compression, we use a special lossy-codec for animations. We also use FSE encoding, more information can be found here .
Video and sounds
There are no general recommendations here; in general, it all comes down to fine-tuning the encoding formats. We cannot get any phenomenal winnings here, we managed to compress the video by about 30% compared to the original version and save about 10-20% on sounds. It all depends on the specific video and audio - you need to select the settings experimentally.
Fonts
Typically, fonts do not take up much space in most applications. But this is until you come across language localizations with hieroglyphic writing. The number of hieroglyphs there can be measured in thousands. This increases the font files up to several megabytes. To combat this, we use an automated script that lists the characters based on the game text file — all unused characters are removed from all fonts.
Text resources and text configuration files
This is usually the smallest group of resources, and it does not need to be separately optimized. The ipa file itself is an archive, and the text data is well archived. In our project, however, the texts take up quite a lot of space. Mainly due to the need to support many localizations.
If all opportunities are exhausted
The size of the build is really a problem for us. How serious it is, you can understand by some additional methods of compression that we have invented.
- Inside the ipa file, the application is packaged in a so-called package. All application files are located in the Payload / <package_name.app> folder. When a file is packed into a zip archive, its full path is added to the table of contents. Therefore, <package_name.app> will be in the table of contents of all game files. And if there are many files, then whatever one may say, and the extra information is duplicated. We renamed the package so that its name consisted of exactly one letter, and at least slightly, but reduced the size.
- Looking through the contents of the executable file, we saw that there are full paths to the original C ++ files. Where did they come from? It turns out that some macros __ FILE __, __ LINE __, got into the production-code, and the compiler carefully opened them for each source file where they were used. We conducted an audit of such macros, and slightly reduced the build.
- Anyone else have any ideas? Write in the comments.
Total
Build size optimization is an extensive topic, this article describes only general principles and methods of resource compression. In addition, there are usually other difficult questions:
- Resource management: how to get rid of duplicate resources and find all unused resources;
- System download resources from the Internet, if we do not climbed into the 100 MB. How to organize the reloading of resources so that the user has a minimum of discomfort, and it would be convenient for designers to work with it;
- How to minimize the consumption of traffic, especially when updating the version of the game.
The techniques listed in the article are the experience of our development team in a particular project. But they are quite versatile, and they can be used not only for mobile games, but also for other applications.
Post scriptum
While the article was being prepared for publication, Apple increased the maximum build to 150 MB. It would seem to be better. But the Google Play limit is still 100 MB. As a result, everything turns out even harder. If we are in the preparation of the build for iOS, we will focus only on the restriction of the Apple store to 150 MB, then after preparing the build for Android we just can’t meet the 100.
You need to make different builds for iOS and Android in order to make the best use of the capabilities of the app stores. Now we are still in the process of finding the best solution. And we are looking forward to a similar step to increase the limit from Google. After that, you can simply replace the number 100 by 150 in the text of the article, and everything will be correct again in it.
Source: https://habr.com/ru/post/342202/
All Articles