Introduction to IL2CPP
Unity continues to improve the IL2CPP technology, and we publish a translation of an article on how it works.
About a year ago we wrote about the future of scripting in Unity. The new scripting technology IL2CPP was supposed to provide the engine with a high-performance portable virtual machine. In January, we released our first platform on 64-bit iOS , using IL2CPP. With the release of Unity 5, another platform has appeared - WebGL . With the support of a huge community of users, we have released many patches and updates for the IL2CPP, gradually optimizing the compiler and increasing the speed of the environment.
')
In the meantime, we continue to improve the technology IL2CPP, it would be nice to talk about how it works. We plan to write a series of articles on such topics:
1. Basics - a set of tools and command line arguments (this article).
2. Excursion on the generated code.
3. Tips for debugging generated code.
4. Method calls: ordinary methods, virtual methods and others.
5. Implementation of the general exchange.
6. P / Invoke wrappers for types and methods.
7. Integrate the garbage collector.
8. Testing and application of frameworks.
In these articles we will discuss some features of the implementation of IL2CPP. I hope this information is useful to you.
What is IL2CPP?
The IL2CPP technology consists of two parts:
• Ahead-of-time compiler (AOT);
• an executable library to support the virtual machine.
The AOT compiler translates an intermediate language (IL) from .NET compilers into C ++ source code. The executable library provides services and abstractions (such as a garbage collector), cross-platform access to streams and files, as well as methods for implementing internal calls (unmanaged code that directly modifies the managed data structures).
AOT compiler
The AOT compiler for IL2CPP is called il2cpp.exe. On Windows, you can find it in the Editor \ Data \ il2cpp directory, and on OS X, in the Contents / Frameworks / il2cpp / build directory in the Unity installation location. The il2cpp.exe utility is written entirely in C # and compiled using .NET and Mono compilers.
This utility accepts managed assemblies compiled by the Mono compiler shipped with Unity and generates the C ++ code that we pass to the C ++ compiler for a specific platform.
The IL2CPP toolbox can be schematically represented as follows:
Executable library
The second part of the IL2CPP technology is an executable library to support the virtual machine. We wrote this library almost entirely in C ++ (there is some platform code in it, but let it stay between us) and called it libil2cpp. It comes in the form of a static library associated with the player's executable file, and its simplicity and portability are among the key advantages of the IL2CPP technology.
You can get a clearer picture of the organization of the libil2cpp code by looking at the header files supplied with Unity (you can find them in the Editor \ Data \ PlaybackEngines \ webglsupport \ BuildTools \ Libraries \ libil2cpp \ include directory in Windows, or in the Contents / Frameworks / directory il2cpp / libil2cpp - on OS X). For example, the interface between C ++ code generated by il2cpp.exe and libil2cpp is in the header file codegen / il2cpp-codegen.h.
One of the key elements of the environment is the garbage collector. Unity 5 comes with libgc, a Boehm-Demers-Weiser garbage collector. However, libil2cpp supports other collectors. For example, we are considering the possibility of integration with Microsoft GC - an open source assembler in the CoreCLR bundle. In one of the following articles we will tell about it in more detail.
How to run il2cpp.exe?
Let's look at an example. For this, I will create a new empty project in Unity 5.0.1 on Windows. So that we have at least one user script, I will add a simple component MonoBehaviour to the main camera:
When building for the WebGL platform, I can use Process Explorer to see the command to run il2cpp.exe:
This is a fairly long string, so let's break it down in parts. First, Unity runs this executable file:
"C: \ Program Files \ Unity \ Editor \ Data \ MonoBleedingEdge \ bin \ mono.exe"
The next argument is the il2cpp.exe utility itself.
"C: \ Program Files \ Unity \ Editor \ Data \ il2cpp / il2cpp.exe"
The remaining arguments are passed to il2cpp.exe, not mono.exe. Let's take them apart. First, Unity passes 5 flags of il2cpp.exe:
• -copy-level = None
Indicates that il2cpp.exe should not perform special copies of the files generated by C ++ code.
• -enable-generic-sharing
Reduces the size of the code and binary file. Over time, we will publish the process of executing generic methods in IL2CPP.
• -enable-unity-event-support
Provides correct code generation for Unity events triggered by reflection.
• -output-format = Compact
Generates C ++ code in a format with fewer characters for type names and methods. Given that the names in the intermediate language are not saved, such code is harder to debug, but it does compile faster because the C ++ compiler needs to parse less code.
• -extra-types.file = "C: \ Program Files \ Unity \ Editor \ Data \ il2cpp \ il2cpp_default_extra_types.txt"
Uses standard empty file for additional types. You can add it to a Unity project so that il2cpp.exe knows which generic or indexable types are created at runtime, but are not listed in the IL code.
It is worth noting that this set of command line arguments for il2cpp.exe is still unstable and is likely to change in future versions.
So, the command line contains 2 files and 1 directory:
• “C: \ Users \ Josh Peterson \ Documents \ IL2CPP Blog Example \ Temp \ StagingArea \ Data \ Managed \ Assembly-CSharp.dll”
• “C: \ Users \ Josh Peterson \ Documents \ IL2CPP Blog Example \ Temp \ StagingArea \ Data \ Managed \ UnityEngine.UI.dll”
• “C: \ Users \ Josh Peterson \ Documents \ IL2CPP Blog Example \ Temp \ StagingArea \ Data \ il2cppOutput”
The il2cpp.exe utility accepts a list of all IL assemblies to be converted. In our case, this includes my simple MonoBehavior script, Assembly-CSharp.dll and UnityEngine.UI.dll. Please note that there are clearly several assemblies missing. Obviously, my script refers to UnityEngine.dll, which means that at least mscorlib.dll, and possibly other assemblies are needed. But where are they? The fact is that the resolution of assemblies occurs inside il2cpp.exe. They can be mentioned on the command line, but this is optional. It turns out, Unity needs only to specify root assemblies (to which other assemblies do not refer).
The last command line argument, il2cpp.exe, is the directory in which the C ++ files are created. If you're interested, you can familiarize yourself with the generated files, but we'll talk more about them in the next article. Before you open the directory, you should choose the Development Player option in the WebGL assembly settings. This removes the -output-format = Compact argument and improves the mapping of type names and methods in C ++ code.
Try playing with the Player Settings WebGL or iOS settings. You will see various command line parameters passed to il2cpp.exe and are responsible for the different stages of code generation. For example, by setting the Enable Exceptions parameter in WebGL Player Settings to Full, you will add the arguments -emit-null-checks, -enable-stacktrace and -enable-array-bounds-check to the command line.
What does IL2CPP not do?
I would like to draw attention to one difficulty, which we decided to bypass and thus avoided many problems. We did not attempt to rewrite the standard C # library under IL2CPP. When you create a Unity project using IL2CPP, all the code in the standard C # libraries (mscorlib.dll, System.dll, etc.) is an exact copy of the Mono script code.
We use the code of standard C # libraries, because it has already managed to establish itself well in many projects. Thus, if an error occurs somewhere, we can be sure that the problem is in the AOT compiler or the executable library.
Development, testing and release of IL2CPP
Since the first official release of IL2CPP version 4.6.1p5 in January, we released 6 full updates and 7 patches (for Unity 4.6 and 5.0), in which we fixed more than a hundred bugs.
To ensure continuous optimization, we are developing updates based on a single version of the code. Before each release, we transfer all changes to a specific release branch, conduct testing and check whether all errors have been successfully corrected. Our QA and Sustained Engineering teams are doing their utmost to keep things going at this pace, and bugs are fixed every week.
Our community has provided us with invaluable assistance by providing many useful error messages. We are very grateful to users for all the reviews that help to continuously improve IL2CPP.
For IL2CPP developers, testing comes first. We often take as a basis development techniques through testing and rarely send inclusion requests if the code has not been fully tested. This strategy works well for technologies such as the IL2CPP, where there are clear inputs and outputs. This means that the overwhelming majority of the errors we encounter are special cases rather than unexpected system behavior ( for example , using 64-bit IntPtr as a 32-bit array index can result in a C ++ compiler error). Thus, we can quickly identify and fix any bugs.
With the support of our community, we are making IL2CPP more stable and more productive all the time.
To be continued
I'm afraid I mention the following articles too often. Just want to tell a lot, and it does not fit in one post. Next time we will dig into the code generated by il2cpp.exe and talk about the work of the C ++ compiler.
About a year ago we wrote about the future of scripting in Unity. The new scripting technology IL2CPP was supposed to provide the engine with a high-performance portable virtual machine. In January, we released our first platform on 64-bit iOS , using IL2CPP. With the release of Unity 5, another platform has appeared - WebGL . With the support of a huge community of users, we have released many patches and updates for the IL2CPP, gradually optimizing the compiler and increasing the speed of the environment.
')
In the meantime, we continue to improve the technology IL2CPP, it would be nice to talk about how it works. We plan to write a series of articles on such topics:
1. Basics - a set of tools and command line arguments (this article).
2. Excursion on the generated code.
3. Tips for debugging generated code.
4. Method calls: ordinary methods, virtual methods and others.
5. Implementation of the general exchange.
6. P / Invoke wrappers for types and methods.
7. Integrate the garbage collector.
8. Testing and application of frameworks.
In these articles we will discuss some features of the implementation of IL2CPP. I hope this information is useful to you.
What is IL2CPP?
The IL2CPP technology consists of two parts:
• Ahead-of-time compiler (AOT);
• an executable library to support the virtual machine.
The AOT compiler translates an intermediate language (IL) from .NET compilers into C ++ source code. The executable library provides services and abstractions (such as a garbage collector), cross-platform access to streams and files, as well as methods for implementing internal calls (unmanaged code that directly modifies the managed data structures).
AOT compiler
The AOT compiler for IL2CPP is called il2cpp.exe. On Windows, you can find it in the Editor \ Data \ il2cpp directory, and on OS X, in the Contents / Frameworks / il2cpp / build directory in the Unity installation location. The il2cpp.exe utility is written entirely in C # and compiled using .NET and Mono compilers.
This utility accepts managed assemblies compiled by the Mono compiler shipped with Unity and generates the C ++ code that we pass to the C ++ compiler for a specific platform.
The IL2CPP toolbox can be schematically represented as follows:
Executable library
The second part of the IL2CPP technology is an executable library to support the virtual machine. We wrote this library almost entirely in C ++ (there is some platform code in it, but let it stay between us) and called it libil2cpp. It comes in the form of a static library associated with the player's executable file, and its simplicity and portability are among the key advantages of the IL2CPP technology.
You can get a clearer picture of the organization of the libil2cpp code by looking at the header files supplied with Unity (you can find them in the Editor \ Data \ PlaybackEngines \ webglsupport \ BuildTools \ Libraries \ libil2cpp \ include directory in Windows, or in the Contents / Frameworks / directory il2cpp / libil2cpp - on OS X). For example, the interface between C ++ code generated by il2cpp.exe and libil2cpp is in the header file codegen / il2cpp-codegen.h.
One of the key elements of the environment is the garbage collector. Unity 5 comes with libgc, a Boehm-Demers-Weiser garbage collector. However, libil2cpp supports other collectors. For example, we are considering the possibility of integration with Microsoft GC - an open source assembler in the CoreCLR bundle. In one of the following articles we will tell about it in more detail.
How to run il2cpp.exe?
Let's look at an example. For this, I will create a new empty project in Unity 5.0.1 on Windows. So that we have at least one user script, I will add a simple component MonoBehaviour to the main camera:
When building for the WebGL platform, I can use Process Explorer to see the command to run il2cpp.exe:
"C:\Program Files\Unity\Editor\Data\MonoBleedingEdge\bin\mono.exe" "C:\Program Files\Unity\Editor\Data\il2cpp/il2cpp.exe" --copy-level=None --enable-generic-sharing --enable-unity-event-support --output-format=Compact --extra-types.file="C:\Program Files\Unity\Editor\Data\il2cpp\il2cpp_default_extra_types.txt" "C:\Users\Josh Peterson\Documents\IL2CPP Blog Example\Temp\StagingArea\Data\Managed\Assembly-CSharp.dll" "C:\Users\Josh Peterson\Documents\IL2CPP Blog Example\Temp\StagingArea\Data\Managed\UnityEngine.UI.dll" "C:\Users\Josh Peterson\Documents\IL2CPP Blog Example\Temp\StagingArea\Data\il2cppOutput"
This is a fairly long string, so let's break it down in parts. First, Unity runs this executable file:
"C: \ Program Files \ Unity \ Editor \ Data \ MonoBleedingEdge \ bin \ mono.exe"
The next argument is the il2cpp.exe utility itself.
"C: \ Program Files \ Unity \ Editor \ Data \ il2cpp / il2cpp.exe"
The remaining arguments are passed to il2cpp.exe, not mono.exe. Let's take them apart. First, Unity passes 5 flags of il2cpp.exe:
• -copy-level = None
Indicates that il2cpp.exe should not perform special copies of the files generated by C ++ code.
• -enable-generic-sharing
Reduces the size of the code and binary file. Over time, we will publish the process of executing generic methods in IL2CPP.
• -enable-unity-event-support
Provides correct code generation for Unity events triggered by reflection.
• -output-format = Compact
Generates C ++ code in a format with fewer characters for type names and methods. Given that the names in the intermediate language are not saved, such code is harder to debug, but it does compile faster because the C ++ compiler needs to parse less code.
• -extra-types.file = "C: \ Program Files \ Unity \ Editor \ Data \ il2cpp \ il2cpp_default_extra_types.txt"
Uses standard empty file for additional types. You can add it to a Unity project so that il2cpp.exe knows which generic or indexable types are created at runtime, but are not listed in the IL code.
It is worth noting that this set of command line arguments for il2cpp.exe is still unstable and is likely to change in future versions.
So, the command line contains 2 files and 1 directory:
• “C: \ Users \ Josh Peterson \ Documents \ IL2CPP Blog Example \ Temp \ StagingArea \ Data \ Managed \ Assembly-CSharp.dll”
• “C: \ Users \ Josh Peterson \ Documents \ IL2CPP Blog Example \ Temp \ StagingArea \ Data \ Managed \ UnityEngine.UI.dll”
• “C: \ Users \ Josh Peterson \ Documents \ IL2CPP Blog Example \ Temp \ StagingArea \ Data \ il2cppOutput”
The il2cpp.exe utility accepts a list of all IL assemblies to be converted. In our case, this includes my simple MonoBehavior script, Assembly-CSharp.dll and UnityEngine.UI.dll. Please note that there are clearly several assemblies missing. Obviously, my script refers to UnityEngine.dll, which means that at least mscorlib.dll, and possibly other assemblies are needed. But where are they? The fact is that the resolution of assemblies occurs inside il2cpp.exe. They can be mentioned on the command line, but this is optional. It turns out, Unity needs only to specify root assemblies (to which other assemblies do not refer).
The last command line argument, il2cpp.exe, is the directory in which the C ++ files are created. If you're interested, you can familiarize yourself with the generated files, but we'll talk more about them in the next article. Before you open the directory, you should choose the Development Player option in the WebGL assembly settings. This removes the -output-format = Compact argument and improves the mapping of type names and methods in C ++ code.
Try playing with the Player Settings WebGL or iOS settings. You will see various command line parameters passed to il2cpp.exe and are responsible for the different stages of code generation. For example, by setting the Enable Exceptions parameter in WebGL Player Settings to Full, you will add the arguments -emit-null-checks, -enable-stacktrace and -enable-array-bounds-check to the command line.
What does IL2CPP not do?
I would like to draw attention to one difficulty, which we decided to bypass and thus avoided many problems. We did not attempt to rewrite the standard C # library under IL2CPP. When you create a Unity project using IL2CPP, all the code in the standard C # libraries (mscorlib.dll, System.dll, etc.) is an exact copy of the Mono script code.
We use the code of standard C # libraries, because it has already managed to establish itself well in many projects. Thus, if an error occurs somewhere, we can be sure that the problem is in the AOT compiler or the executable library.
Development, testing and release of IL2CPP
Since the first official release of IL2CPP version 4.6.1p5 in January, we released 6 full updates and 7 patches (for Unity 4.6 and 5.0), in which we fixed more than a hundred bugs.
To ensure continuous optimization, we are developing updates based on a single version of the code. Before each release, we transfer all changes to a specific release branch, conduct testing and check whether all errors have been successfully corrected. Our QA and Sustained Engineering teams are doing their utmost to keep things going at this pace, and bugs are fixed every week.
Our community has provided us with invaluable assistance by providing many useful error messages. We are very grateful to users for all the reviews that help to continuously improve IL2CPP.
For IL2CPP developers, testing comes first. We often take as a basis development techniques through testing and rarely send inclusion requests if the code has not been fully tested. This strategy works well for technologies such as the IL2CPP, where there are clear inputs and outputs. This means that the overwhelming majority of the errors we encounter are special cases rather than unexpected system behavior ( for example , using 64-bit IntPtr as a 32-bit array index can result in a C ++ compiler error). Thus, we can quickly identify and fix any bugs.
With the support of our community, we are making IL2CPP more stable and more productive all the time.
To be continued
I'm afraid I mention the following articles too often. Just want to tell a lot, and it does not fit in one post. Next time we will dig into the code generated by il2cpp.exe and talk about the work of the C ++ compiler.
Source: https://habr.com/ru/post/276589/
All Articles