The problem with reading files more than 1Mb in Android

Good day.

When writing a single Android application, I ran into the problem of downloading files from the assets directory which are larger than 1Mb.
Suppose, in the project folder, in assets there is a file file.dat , which needs to be loaded into memory and read data from it.


')
Open the file, and read in buffer 8 first bytes

AssetManager am = context.getAssets();
InputStream is = am.open("file.dat", AssetManager.ACCESS_BUFFER);
// Read file
is.read(buffer, 0, 8);

As a result of code execution on Android 2.2 and below, we receive a message in LogCat
06-14 17: 10: 48.554: DEBUG / asset (382): Data exceeds UNCOMPRESS_DATA_MAX (1048852 vs 1048576)
and IOException
06-14 17: 10: 48.654: WARN / System.err (698): java.io.IOException

We look at the source of the android. In the frameworks / base / include / utils / Asset.h file there is the same constant UNCOMPRESS_DATA_MAX

enum {
/* data larger than this does not get uncompressed into a buffer */
#ifdef HAVE_ANDROID_OS
UNCOMPRESS_DATA_MAX = 1 * 1024 * 1024
#else
UNCOMPRESS_DATA_MAX = 2 * 1024 * 1024
#endif
};

And in the frameworks / base / include / utils / Asset.cpp file there is a check for the file size

if (mUncompressedLen > UNCOMPRESS_DATA_MAX) {
LOGD("Data exceeds UNCOMPRESS_DATA_MAX (%ld vs %d)\n",
(long) mUncompressedLen, UNCOMPRESS_DATA_MAX);
goto bail;
}

As can be seen from the comment, data larger than UNCOMPRESS_DATA_MAX in bytes is not unpacked into the buffer from the .apk file.

There are several ways to solve the problem:
1. The easiest way is to change the file.dat file extension to such that this file will not be compressed when packed into an .apk file. A list of these extensions is defined in Package.cpp

/* these formats are already compressed, or don't compress well */
static const char* kNoCompressExt[] = {
".jpg", ".jpeg", ".png", ".gif",
".wav", ".mp2", ".mp3", ".ogg", ".aac",
".mpg", ".mpeg", ".mid", ".midi", ".smf", ".jet",
".rtttl", ".imy", ".xmf", ".mp4", ".m4a",
".m4v", ".3gp", ".3gpp", ".3g2", ".3gpp2",
".amr", ".awb", ".wma", ".wmv"
};

Alternatively, you can manually compile the project using aapt with the "-0" option, specifying file extensions that should not be compressed in .apk . If you use an empty string instead of an extension, no file will be compressed.
This method is bad because the size of the downloadable installation file from the market increases significantly (by the difference between compressed and uncompressed file.dat ), and the size of the file is quite critical for mobile devices. In my case, the text information used in the application is stored in file.dat , in the unpacked form it takes a little more than 1Mb, and in the .apk file it is compressed to 40Kb.
If your files from assets are badly compressed in zip (media), this method is perfect.

2. The second way is to split file.dat into 1MB pieces, for example from the console:
split file.dat --bytes=1024K

The resulting files need to be placed in the assets , when you first start the application, "glue" together and put in the directory with the application. This method is well described on stackoverflow, it was proposed by a person under the name Seva Alekseyev, so I’ll only give the link stackoverflow.com/questions/2860157/load-files-bigger-than-1m-from-assets-folder/3093966#3093966
In this case, your application will store two copies of the data, one in the .apk file, the other, uploaded to the directory with the application. It also adds a bit of extra code and the need to monitor the existence and integrity of the "unpacked" data.

The described problem for Android 2.3.1 and above is not.

Of course, you can think of something else, for example, downloading files over the network (the so-called cache), which is not always reliable if the user turned off the Internet, or generating a file from scratch, but I had the two methods described above.