Implementing a Dynamically Downloadable Content (DLC) System for a Unity 3D Mobile Game
Recently, for one game on Unity 3D, which we developed, it became necessary to add a DLC system . Although it was not so easy as it seemed at the beginning, we successfully coped with the problems that had arisen and the game went into gold. In this article I want to present our version of the implementation of the DLC, to talk about the problems that have arisen and how we solved them.
The game has a store where the player buys things for game or real currency. In the store - more than 200 things. When a player enters the game, 20 items are available in the store. If there is Internet, the game without the knowledge of the user polls the server for the presence of a DLC and, if there is one, downloads it in the background. When a player re-enters the store, he will see all the new things from the DLC.
There is also a set of locations. Each location has a set of textures and .asset files. New locations must also be added via DLC.
Downloading resources from the DLC should be synchronous.
Platform: iOS (iPhone 3GS and above.) And Android (Samsung Galaxy S and above).
In the game, things are completely determined by the itemdata.txt file, which contains information about things and their textures. This means that in each DLC there will be an itemdata.txt file with a set of those things that are in the DLC + logs for these things. And when the store requests a database of things, we glue all the text files from all the DLC and give it this file.
Similarly, for locations there is a file locationdata.txt with a list and characteristics of locations + textures and asset files for them.
The corresponding C # code for loading resources in the game logic will look like this:
')
Similarly, when requesting a texture, we check its presence in the DLC. If it is there, we download, otherwise we download from game resources. If there is not, then we load something default.
Similarly, the .asset files will have the function GetAsset (string assetName). Its implementation will be similar, so skip it.
We decided that we should have in the DLC. It remains to decide in what form it is all stored.
The first option is to store the DLC as a zip archive. In each archive - a text file + N textures. Textures must be in PVRTC format to save video memory. But here we have the first problem - Unity only supports loading textures from the file system in PNG or JPG [ link ] format. Then the texture can be written to the PVRTC texture [ link ]. This is a slow process, because requires conversion to PVR in realtime. Moreover, since the DLC plans to store files of the .asset type, and possibly game levels (.scene), such a method is completely unsuitable.
The second option is to use AssetBundle . This solution is ideal for DLC in games.
Judging by the documentation, it has a lot of advantages:
From minuses only that AssetBundle demands Pro version of Unity and does not support enciphering . We decided to stay on this decision, because it is obviously more attractive and allows us to solve all our problems.
To begin with, a test version of the DLC system with the most elementary functionality was made.
At first, all 200 or more store item textures and location files were packed into one AssetBundle and uploaded to the server. The file turned out about 200 MB. Packaging in AssetBundle was performed by a script in the editor. How to make the packaging of resources in AssetBundle is well described in the documentation. You can also use my script to create an AssetBundle .
Next, after starting the game, we do the following steps:
The above code is still the only possible way to load a resource synchronously from AssetBundle. And as it turned out, there are a lot of nuances. Let's sort them in order.
The
The reader may wonder why not load AssetBundle asynchronously, for example, with the LoadFromCacheOrDownload function, and then just take the necessary resources from it synchronously. After all, it is logical that AssetBundle, when loading from the file system, should load only the file header, and therefore should be doing a bit of memory in memory.
However, this was not the case. Loaded AssetBundle is stored in memory completely with all its contents in the unpacked form. Thus, to load one texture from 200, Unity will load all 200 textures into memory, take one, and then free up memory for the remaining 199 textures. We found this experimentally by measuring the memory on the device.
Obviously, this is unacceptable for mobile devices.
The given variant is the only way we found to implement synchronous loading of DLC and resources from it.
Uncompressed AsssetBundle is required, which leads to a large loss of time and traffic when loading a DLC.
The option is suitable for relatively small AssetBundles. consumes a lot of RAM.
Let's try to take into account all previous problems and find solutions for them.
The problem with loading large assetBundles can be solved in two ways.
The first is to use the WebClient class. However, we have problems with it on iOS. WebClient could not download anything, but it worked fine on the desktop.
The second option is to use native OS functions. For example, NSURLConnection for iOS and URLConnection for Android, respectively, which support buffered download directly to a disk file.
But this is not such a big problem, because in any case, we need to reduce the size of AssetBundle for synchronous loading. Therefore, for the time being we have left the current method of loading bundles from the server.
A much more serious problem is the synchronous download of AssetBundle. Since it should not only be uncompressed, but also take up little space in the memory; in one way or another, we have to split our one large DLC file into many small files. However, if we split into too small files, there will be many of them and this will greatly increase the load time, because it is necessary to establish a new connection for each file. So, we still have to keep them compressed to better save load time and traffic.
To solve this problem, it was decided to use its own archiver. An open archiver library for C # was chosen, which was effortlessly brought under Mono into Unity.
Further, the algorithm of actions was as follows:
Here we have another problem. Since we now use a bundle compressed by the archiver, we can no longer download it with the LoadFromCacheOrDownload function. So, now we have to define our own version control system for DLC.
For the DLC version control system, the following solution was chosen. On the server in the folder where the DLC files were located, the text file dlcversion was created. It contained a list of DLCs in the folder and md5 hashes for them. These hashes were counted at the stage of the DLC aplode to the server. The client had the same exact file, and when the application started, the client compared its file with the file on the server. If some DLC file had excellent hashes or there was no hash at all, it was considered that the file on the client was outdated and the client was pulling a new DLC file from the server.
After the new DLC file was downloaded and unpacked, its hash was once again checked against the server one, and only after that the outdated file was replaced with a new one and the corresponding entry was made in the client's dlcversion file.
The described system was successfully implemented and works fine. The only drawback we had was a slight drawdown of fps (lag) when downloading and unpacking the DLC in the background. And also the peak values ​​of the memory consumption of the application have slightly increased.
Thanks for attention. I will be glad to answer your questions.
Formulation of the problem
The game has a store where the player buys things for game or real currency. In the store - more than 200 things. When a player enters the game, 20 items are available in the store. If there is Internet, the game without the knowledge of the user polls the server for the presence of a DLC and, if there is one, downloads it in the background. When a player re-enters the store, he will see all the new things from the DLC.
There is also a set of locations. Each location has a set of textures and .asset files. New locations must also be added via DLC.
Downloading resources from the DLC should be synchronous.
Platform: iOS (iPhone 3GS and above.) And Android (Samsung Galaxy S and above).
DLC content and work with it in the game
In the game, things are completely determined by the itemdata.txt file, which contains information about things and their textures. This means that in each DLC there will be an itemdata.txt file with a set of those things that are in the DLC + logs for these things. And when the store requests a database of things, we glue all the text files from all the DLC and give it this file.
Similarly, for locations there is a file locationdata.txt with a list and characteristics of locations + textures and asset files for them.
The corresponding C # code for loading resources in the game logic will look like this:
public String GetItemDataBase() { if(DLCManager.isDLCLoaded() == true) { // itemdata.txt DLC string String itemListStr = DLCManager.GetTextFileFromAllDLCs(“itemdata”); return itemListStr; } else { // TextAsset itemTextFile = Resources.Load(“itemdata”) as TextAsset; return itemTextFile.text; } return String.Empty; } ')
Similarly, when requesting a texture, we check its presence in the DLC. If it is there, we download, otherwise we download from game resources. If there is not, then we load something default.
public Texture GetTexture(string txname) { Texture tx = null; if(DLCManager.isDLCLoaded() == true) { tx = DLCManager.GetTextureFromDLC(txname); } if(tx == null) { tx = Resources.Load(txname) as Texture; } if(tx == null) { Assert(tx, “Texture not find: ” + txname); tx = Resources.Load(kDefaultItemTexturePath) as Texture; } return tx; } Similarly, the .asset files will have the function GetAsset (string assetName). Its implementation will be similar, so skip it.
DLC file
We decided that we should have in the DLC. It remains to decide in what form it is all stored.
The first option is to store the DLC as a zip archive. In each archive - a text file + N textures. Textures must be in PVRTC format to save video memory. But here we have the first problem - Unity only supports loading textures from the file system in PNG or JPG [ link ] format. Then the texture can be written to the PVRTC texture [ link ]. This is a slow process, because requires conversion to PVR in realtime. Moreover, since the DLC plans to store files of the .asset type, and possibly game levels (.scene), such a method is completely unsuitable.
The second option is to use AssetBundle . This solution is ideal for DLC in games.
Judging by the documentation, it has a lot of advantages:
- It can store any Unity resources, including compressed in the right texture format (what we need).
- This is an archive with good compression.
- Easy and convenient to use.
- Supports the version parameter and the hash sum (when loaded by the LoadFromCacheOrDownload function), which is convenient for controlling DLC ​​versions
From minuses only that AssetBundle demands Pro version of Unity and does not support enciphering . We decided to stay on this decision, because it is obviously more attractive and allows us to solve all our problems.
Implementation (Option 1)
To begin with, a test version of the DLC system with the most elementary functionality was made.
At first, all 200 or more store item textures and location files were packed into one AssetBundle and uploaded to the server. The file turned out about 200 MB. Packaging in AssetBundle was performed by a script in the editor. How to make the packaging of resources in AssetBundle is well described in the documentation. You can also use my script to create an AssetBundle .
Next, after starting the game, we do the following steps:
- First you need to download the DLC from the server. We do this according to the code from the manual Unity. Next, write the downloaded data to a file on the disk for future use.
// Start a download of the given URL using assetBundle version and CRC-32 Checksum WWW www = WWW.LoadFromCacheOrDownload (urlToAssetBundle, version, crc32Checksum); // Wait for download to complete yield return www; // Get the byte data byte[] byteData = www.bytes; // , byteData = MyDescriptionMethod(byteData); // byteData .unity3d ... // Frees the memory from the web stream www.Dispose(); //DLC DLCManager.SetDLCLoaded(true);
On this code, we are most likely to get crashes from memory on low devices like the iPhone 3GS, since The WWW class does not support buffered loading and stores all loaded information in memory. We will talk about this issue a little later. For now, remember this moment and move on. - Downloading resources from the DLC.
Now we need to define the functions GetTextureFromDLC (), GetAssetFromDLC () and GetTextFileFromAllDLCs (). The definition of the latter is still omitted, since it is almost no different from the first except for the type of resource being loaded.
The main task of the GetTextureFromDLC function is the synchronous loading of a texture by name from the DLC.
Let's try to define it as follows.public Texture GetTextureFromDLC(String textureName) { // DLC . . AssetBundle asset = AssetBundle.CreateFromFile(pathToAssetBundle); // DLC Texture texture = asset.Load(textureName) as Texture; // texture asset.Unload(false); return texture; }
The above code is still the only possible way to load a resource synchronously from AssetBundle. And as it turned out, there are a lot of nuances. Let's sort them in order.
The
AssetBundle.CreateFromFile function according to the documentation synchronously loads the asset from the disk. But there is one caveat - “Only uncompressed asset bundles are supported by this function.” Thus, only uncompressed AssetBundle can be loaded synchronously. This will significantly increase traffic and DLC download time from the server. In addition, Unity does not support converting AssetBundle from compressed to uncompressed, so you will not be able to download the compressed bundle, and then unpack it on the client.The reader may wonder why not load AssetBundle asynchronously, for example, with the LoadFromCacheOrDownload function, and then just take the necessary resources from it synchronously. After all, it is logical that AssetBundle, when loading from the file system, should load only the file header, and therefore should be doing a bit of memory in memory.
However, this was not the case. Loaded AssetBundle is stored in memory completely with all its contents in the unpacked form. Thus, to load one texture from 200, Unity will load all 200 textures into memory, take one, and then free up memory for the remaining 199 textures. We found this experimentally by measuring the memory on the device.
Obviously, this is unacceptable for mobile devices.
Summary
The given variant is the only way we found to implement synchronous loading of DLC and resources from it.
Uncompressed AsssetBundle is required, which leads to a large loss of time and traffic when loading a DLC.
The option is suitable for relatively small AssetBundles. consumes a lot of RAM.
Bug fixes (Option 2)
Let's try to take into account all previous problems and find solutions for them.
The problem with loading large assetBundles can be solved in two ways.
The first is to use the WebClient class. However, we have problems with it on iOS. WebClient could not download anything, but it worked fine on the desktop.
The second option is to use native OS functions. For example, NSURLConnection for iOS and URLConnection for Android, respectively, which support buffered download directly to a disk file.
But this is not such a big problem, because in any case, we need to reduce the size of AssetBundle for synchronous loading. Therefore, for the time being we have left the current method of loading bundles from the server.
A much more serious problem is the synchronous download of AssetBundle. Since it should not only be uncompressed, but also take up little space in the memory; in one way or another, we have to split our one large DLC file into many small files. However, if we split into too small files, there will be many of them and this will greatly increase the load time, because it is necessary to establish a new connection for each file. So, we still have to keep them compressed to better save load time and traffic.
To solve this problem, it was decided to use its own archiver. An open archiver library for C # was chosen, which was effortlessly brought under Mono into Unity.
Further, the algorithm of actions was as follows:
- When creating the bundle, the option BuildOptions.UncompAsAssetBundle was specified to get an uncompressed bundle.
- Then the bundle was archived and encrypted by the archiver and uploaded to the server.
- While the application was running, a separate stream was created, which in the background pumped out bundles, unpacked them and put them into a special folder.
Here we have another problem. Since we now use a bundle compressed by the archiver, we can no longer download it with the LoadFromCacheOrDownload function. So, now we have to define our own version control system for DLC.
For the DLC version control system, the following solution was chosen. On the server in the folder where the DLC files were located, the text file dlcversion was created. It contained a list of DLCs in the folder and md5 hashes for them. These hashes were counted at the stage of the DLC aplode to the server. The client had the same exact file, and when the application started, the client compared its file with the file on the server. If some DLC file had excellent hashes or there was no hash at all, it was considered that the file on the client was outdated and the client was pulling a new DLC file from the server.
After the new DLC file was downloaded and unpacked, its hash was once again checked against the server one, and only after that the outdated file was replaced with a new one and the corresponding entry was made in the client's dlcversion file.
The described system was successfully implemented and works fine. The only drawback we had was a slight drawdown of fps (lag) when downloading and unpacking the DLC in the background. And also the peak values ​​of the memory consumption of the application have slightly increased.
Thanks for attention. I will be glad to answer your questions.
Source: https://habr.com/ru/post/188824/
All Articles