Icy Rocks Test for Android: it's time for a real test
Android OS, if you compare it with Windows, Linux or OSX, the system is young. However, it is used by more than a billion people. How exactly do they work with Android? How to make the system faster and more convenient? The novelty of the platform does not allow to give clear answers to these questions. Moreover, user interaction scenarios with Android devices and applications are constantly changing, often quite strongly and unexpectedly.
To study the peculiarities of user interaction with various software and hardware complexes, special tests are created, the so-called options or scenarios for using systems (use cases). They are also called “workloads” (workloads). All this is a model of real-life work scenarios, they are not real applications. When applied to Android, such tests can be used to measure and improve performance.
Unfortunately, for Android, there are only a few applications that plausibly mimic the real workload. The tests used today, like CaffeineMark, either have nothing to do with these scenarios, or are too simple. These tests simply do not contain tests aimed at exploring how the devices are actually used. However, they, by virtue of prevalence, strongly influence public opinion, on how the general public perceives certain devices.
As a result, developers, in terms of the results of such tests, are not able to figure out what sensations the work with this or that device will cause from a real user. If you optimize the system, relying on such tests, you can achieve impressive performance, which is called “on paper” and leave open the main question: “What will it give to users?”.
')
Most of the tests for Android explore the system in one of two main areas. Some focus on a limited set of hardware features of the devices. They are designed to get some data on specific components or sets of components of SoC-platforms. Others are not tied to hardware. They focus on the Java environment level, in fact, using a limited number of functions. Both are synthetic workloads, they do not do the same thing that happens in the code of a typical real application.
The application execution stack in Android is complex, so when analyzing the performance of this system, it is worthwhile to move away from outdated, narrowly focused methods and switch to new ones. For those that more accurately reflect the behavior of people. If our goal is to significantly improve the perception of applications by humans, we need tests that implement realistic scenarios for working with the system. Namely, it is necessary to take into account how the user works with the application, and how the application interacts with the operating system, and, ultimately, with the hardware.
Icy Rocks Workload is a graphical application developed by Intel that simulates a real game on the Android platform. The project uses a physical open source library JBox2D and graphics engine Cocos2D . OpenGL is responsible for displaying graphics in Cocos2D. JBox2D deals with the simulation of moving objects. It built the physics of the game world, he is responsible for updating the positions of objects and their interaction.
The test measures several indicators. The first is the average number of frames per second (Frames Per Second, FPS). FPS is usually used to evaluate the smoothness of graphics output. The average number of animations per second (Animation Per Second, APS) allows you to evaluate the performance of the physical engine. Other indicators are the average frame update time (Frame Update Time), the number of frames dropped per second (Janks Per Second, JPS). The total data collected by the program are displayed on the screen and recorded in a log file on the device.
Icy Rocks in action
The main screen of Icy Rocks is full of sprites. Here are ice floes, snow flakes, snowman and catapult. Catapult throws chunks of ice into a snowman. They are separated by a gorge, snowflakes and ice floes roll there. In the lower part of the gorge there is a rotating agitator, which does not allow to miss all that got to the bottom. The load that the game creates on the system during the simulation gradually increases. This is achieved by adding pieces of ice and snowflakes.
Icy Rocks can be performed in two modes. The first is the demo mode, the second is for testing. In the demo mode, the user can add to the screen floes and snow flakes by touching the screen. In test mode, everything is done automatically. Chunks of ice and snowflakes are added every 20 seconds with a 2: 5 ratio, the program does not respond to touching the screen. A total of five passes of the test are performed, after which the final results are displayed.
Icy Rocks exists in three versions. The first is for the Android platform, the second is for Oracle Java (JDK 8), the third is represented by the native C ++ application. Appearance and animation in all three options are similar.
As already mentioned, there are two modes of the program - demo and test. In demo mode, the runtime is unlimited, in test mode it is about 11 minutes.
Icy Rocs for the Android platform is a reference implementation. Icy Rocks for Java can be run on any PC running Linux with Java 8 installed. Icy Rocks uses native C ++ version of Box2D and Cocos2D-X .
Icy Rocks for Android and Java can be run both in graphics mode and in CPU mode (non-GL). The last option is also called Icy Rocks Workload for Kernel. Its use allows us to estimate the autonomous performance of the physics engine.
Icy Rocks for Android comes as a single GameWorkload.apk package. After installing and running the application, you can select the test mode (benchmark) or demo (demo). In test mode, everything is done automatically, at the end of the work the results are reported. In demo mode, the user can interact with the program.
In a typical real Android game, computational complexity increases when the user reaches a higher level and scores more points. At Icy Rocks, we easily increase the load by adding more ice and snow — this is shown in the illustration below.
Icy Rocks measures the average number of frames output per second (FPS) under a load of varying intensity, then calculates the geometric average of the indicator at different load levels. The test also measures the number of frames dropped per second (Jank Per Second, JPS) at various points in the animation. The term “Jank” in the Java environment is used to denote “ragged”, “hopping” animation.
The typical duration of one session of the game on Android is about 10 minutes, so the test is designed to run for about 11 minutes. This includes 10-second pauses between test passes. Within one pass, the load on the system, due to the addition of pieces of ice and snowflakes, increases. In the beginning on the screen there is neither one nor the other. Then add 20 ice and 50 snowflakes - and so on. When the load increases to a predetermined limit, a 10-second break is taken and everything repeats. There are five passes of the test, each takes about two minutes. Here is how it looks in the diagram.
Testing scheme
When performing each of the test passes, the above mentioned indicators are calculated. Namely:
The main indicators (primary metrics).
- The number of animations per second.
- Frame frequency.
Additional indicators (secondary metrics).
- Screen refresh time in seconds per frame.
- The number of frames dropped.
After the test is completed, the geometric average is calculated from the measurements made in each of the five passes. The final result is as shown below.
Test results
When using Cocos2D for the Android platform, the game sets in motion the main loop in which the user actions are processed, the game logic is implemented (moving objects, for example), animation and GL commands are prepared, and finally a new frame is drawn.
The scheme of the typical game
JBox2D is a library of simulating the interaction of solids. With its help, you can simulate realistic behavior of game objects. JBox2D is used as a subsystem for the implementation of game logic in Cocos2D.
In games, as a step of the game world, usually use 1/60 of a second. JBox2D is responsible for modeling the physical interactions in the game world.
It all starts with defining all the objects in the game world and installing event listeners. It then initializes the scene from the point of view of the physics engine and prepares callback functions. At each step of the game, when the main loop calls JBox2D to update the world, the positions and speeds of the objects are updated.
The scheme of the physics engine
A typical gaming application has two threads: main (main thread) and graphics output stream (renderer thread). The main thread initializes the application and the game screen, and then starts the main loop. The graphics output stream is called every time a scene is drawn, and it is also responsible for invoking event handlers. Here is some information on these two streams:
Main thread
- Initialization of the application (MainActivity.java).
- Initialization of the game screen (GameLayer.java).
- Initialization of physical objects in JBox2D and sprites in Cocos2D.
- Installation of object collision event listeners.
Graphics output stream
- Standard update function of the game world (update () in GameLayer.java).
- Perform a pitch of the game world.
- Passing through all physically objects and setting the position and angle of rotation for the corresponding sprites.
- The collision event listener (postSolve in GameLayer.java) is used to implement special effects.
- An ice floe breaks into small parts when it collides with massive objects.
- Animated display of points when hit by ice in a snowman.
- Screen touch handler (ccToucherEnded in GameLayer.java) is used to add snowflakes and ice to the screen.
- Function to display statistics (showFPS in CCDirector.java).
Icy Rocks Job Scheme
Above is a diagram of the work cycle drawing. It resembles the main cycle, which we considered above, talking about Cocos2D, but our cycle is a bit more complicated. Each time the scheduler timer is triggered, the next frame is calculated and updated. At this time, callback functions and game logic are processed. For example, JBox2D is called to implement game physics. After that, you can start displaying the image.
When the output is complete, a small waiting period is possible - until the FrameTime time comes. This waiting is necessary in order to ensure a stable frame rate. For example, if an application is capable of issuing 100 FPS, it will, thanks to this approach, slow itself down to the required 60 FPS. And the remaining time can be spent on the implementation of more advanced game logic, data preparation and other similar tasks.
Icy Rocks can be viewed as a single-threaded Android test running on the Android GLTHread stream. During the test, most of the time is spent on executing code from Android Runtime (ART) and working in low-level OpenGL libraries. Some time passes in the machine code of Android, called via the Java Native Interface (JNI) and System.arraycopy.
In the course of the test, in fact, the possibilities of Android Runtime, the graphics capabilities of the Android stack, and the hardware capabilities of the SoC platform are explored.
The Icy Rocks test for Java is demanding for arithmetic logic operations (ALU), the branch prediction subsystem (floating-point comparison), the performance of the associative data translation buffer (DTLB) and the second-level cache (L2).
As already mentioned, the test contains five identical passes. Each of them is divided into 20-second intervals, during which the number of snowflakes and ice floats remains constant. When moving to the next interval, their number increases. Most of these configurations have identical performance characteristics (for example, a similar number of instructions per cycle, the same Java code that is executed most often, and so on). During the test, 32-bit floating point calculations are intensively used - this is the accuracy of collision detection in a physical engine.
During the experiments with Icy Rocks, we found that the physics engine JBox2D uses its own implementation of the methods of the standard library java.lang.Math. It involves huge data structures to support the operation of the Math.sin () and Math.cos () functions. We recommend making changes to the JBox2D code, using the standard java.lang.Math methods for optimal performance.
In addition, it was found that JBox2D joins objects into a pool so that during program execution you do not waste time allocating memory for them and garbage collection. This approach can degrade the performance of the garbage collector, cause heap fragmentation, worsen the locality of references to objects. Our recommendation is to use the standard garbage collector, and not to pool objects. We look forward to further improving the locality of the links and the garbage collector through the use of a memory allocation facility for objects, which is based on the Thread-Local Allocation Buffer (TLAB) in the near future.
Thanks to Android research using Icy Rocks, Intel has contributed to the Android Open Source Project (AOSP) with several enhancements. There have also been many optimizations in the Intel ART code.
We opened the code Icy Rocks . This is part of Intel's strategy to change the approach to measuring the performance of Android devices. Existing synthetic performance tests, for the most part, have nothing to do with reality, do not reflect the features of these applications. The data they give out may be distorted even more by optimization. We, for example, recently "improved" in this way CFBench and Quadrant.
We want to convince the Android community to abandon synthetic tests, far from reality and subject to manipulation, and replace them with more believable workloads. Such as Icy Rocks.
To study the peculiarities of user interaction with various software and hardware complexes, special tests are created, the so-called options or scenarios for using systems (use cases). They are also called “workloads” (workloads). All this is a model of real-life work scenarios, they are not real applications. When applied to Android, such tests can be used to measure and improve performance.
Unfortunately, for Android, there are only a few applications that plausibly mimic the real workload. The tests used today, like CaffeineMark, either have nothing to do with these scenarios, or are too simple. These tests simply do not contain tests aimed at exploring how the devices are actually used. However, they, by virtue of prevalence, strongly influence public opinion, on how the general public perceives certain devices.
As a result, developers, in terms of the results of such tests, are not able to figure out what sensations the work with this or that device will cause from a real user. If you optimize the system, relying on such tests, you can achieve impressive performance, which is called “on paper” and leave open the main question: “What will it give to users?”.
')
Review Icy Rocks
Most of the tests for Android explore the system in one of two main areas. Some focus on a limited set of hardware features of the devices. They are designed to get some data on specific components or sets of components of SoC-platforms. Others are not tied to hardware. They focus on the Java environment level, in fact, using a limited number of functions. Both are synthetic workloads, they do not do the same thing that happens in the code of a typical real application.
The application execution stack in Android is complex, so when analyzing the performance of this system, it is worthwhile to move away from outdated, narrowly focused methods and switch to new ones. For those that more accurately reflect the behavior of people. If our goal is to significantly improve the perception of applications by humans, we need tests that implement realistic scenarios for working with the system. Namely, it is necessary to take into account how the user works with the application, and how the application interacts with the operating system, and, ultimately, with the hardware.
Icy Rocks Workload is a graphical application developed by Intel that simulates a real game on the Android platform. The project uses a physical open source library JBox2D and graphics engine Cocos2D . OpenGL is responsible for displaying graphics in Cocos2D. JBox2D deals with the simulation of moving objects. It built the physics of the game world, he is responsible for updating the positions of objects and their interaction.
The test measures several indicators. The first is the average number of frames per second (Frames Per Second, FPS). FPS is usually used to evaluate the smoothness of graphics output. The average number of animations per second (Animation Per Second, APS) allows you to evaluate the performance of the physical engine. Other indicators are the average frame update time (Frame Update Time), the number of frames dropped per second (Janks Per Second, JPS). The total data collected by the program are displayed on the screen and recorded in a log file on the device.
Game simulation
Icy Rocks in action
The main screen of Icy Rocks is full of sprites. Here are ice floes, snow flakes, snowman and catapult. Catapult throws chunks of ice into a snowman. They are separated by a gorge, snowflakes and ice floes roll there. In the lower part of the gorge there is a rotating agitator, which does not allow to miss all that got to the bottom. The load that the game creates on the system during the simulation gradually increases. This is achieved by adding pieces of ice and snowflakes.
Icy Rocks can be performed in two modes. The first is the demo mode, the second is for testing. In the demo mode, the user can add to the screen floes and snow flakes by touching the screen. In test mode, everything is done automatically. Chunks of ice and snowflakes are added every 20 seconds with a 2: 5 ratio, the program does not respond to touching the screen. A total of five passes of the test are performed, after which the final results are displayed.
Three options for Icy Rocks
Icy Rocks exists in three versions. The first is for the Android platform, the second is for Oracle Java (JDK 8), the third is represented by the native C ++ application. Appearance and animation in all three options are similar.
As already mentioned, there are two modes of the program - demo and test. In demo mode, the runtime is unlimited, in test mode it is about 11 minutes.
Icy Rocs for the Android platform is a reference implementation. Icy Rocks for Java can be run on any PC running Linux with Java 8 installed. Icy Rocks uses native C ++ version of Box2D and Cocos2D-X .
Icy Rocks for Android and Java can be run both in graphics mode and in CPU mode (non-GL). The last option is also called Icy Rocks Workload for Kernel. Its use allows us to estimate the autonomous performance of the physics engine.
Launch of Icy Rocks
Icy Rocks for Android comes as a single GameWorkload.apk package. After installing and running the application, you can select the test mode (benchmark) or demo (demo). In test mode, everything is done automatically, at the end of the work the results are reported. In demo mode, the user can interact with the program.
Automatic load distribution over time
In a typical real Android game, computational complexity increases when the user reaches a higher level and scores more points. At Icy Rocks, we easily increase the load by adding more ice and snow — this is shown in the illustration below.
Icy Rocks measures the average number of frames output per second (FPS) under a load of varying intensity, then calculates the geometric average of the indicator at different load levels. The test also measures the number of frames dropped per second (Jank Per Second, JPS) at various points in the animation. The term “Jank” in the Java environment is used to denote “ragged”, “hopping” animation.
The typical duration of one session of the game on Android is about 10 minutes, so the test is designed to run for about 11 minutes. This includes 10-second pauses between test passes. Within one pass, the load on the system, due to the addition of pieces of ice and snowflakes, increases. In the beginning on the screen there is neither one nor the other. Then add 20 ice and 50 snowflakes - and so on. When the load increases to a predetermined limit, a 10-second break is taken and everything repeats. There are five passes of the test, each takes about two minutes. Here is how it looks in the diagram.
Testing scheme
When performing each of the test passes, the above mentioned indicators are calculated. Namely:
The main indicators (primary metrics).
- The number of animations per second.
- Frame frequency.
Additional indicators (secondary metrics).
- Screen refresh time in seconds per frame.
- The number of frames dropped.
After the test is completed, the geometric average is calculated from the measurements made in each of the five passes. The final result is as shown below.
Test results
The mechanism of Cocos2D for Android (Java)
When using Cocos2D for the Android platform, the game sets in motion the main loop in which the user actions are processed, the game logic is implemented (moving objects, for example), animation and GL commands are prepared, and finally a new frame is drawn.
The scheme of the typical game
The mechanism of JBox2D
JBox2D is a library of simulating the interaction of solids. With its help, you can simulate realistic behavior of game objects. JBox2D is used as a subsystem for the implementation of game logic in Cocos2D.
In games, as a step of the game world, usually use 1/60 of a second. JBox2D is responsible for modeling the physical interactions in the game world.
It all starts with defining all the objects in the game world and installing event listeners. It then initializes the scene from the point of view of the physics engine and prepares callback functions. At each step of the game, when the main loop calls JBox2D to update the world, the positions and speeds of the objects are updated.
The scheme of the physics engine
Game streams
A typical gaming application has two threads: main (main thread) and graphics output stream (renderer thread). The main thread initializes the application and the game screen, and then starts the main loop. The graphics output stream is called every time a scene is drawn, and it is also responsible for invoking event handlers. Here is some information on these two streams:
Main thread
- Initialization of the application (MainActivity.java).
- Initialization of the game screen (GameLayer.java).
- Initialization of physical objects in JBox2D and sprites in Cocos2D.
- Installation of object collision event listeners.
Graphics output stream
- Standard update function of the game world (update () in GameLayer.java).
- Perform a pitch of the game world.
- Passing through all physically objects and setting the position and angle of rotation for the corresponding sprites.
- The collision event listener (postSolve in GameLayer.java) is used to implement special effects.
- An ice floe breaks into small parts when it collides with massive objects.
- Animated display of points when hit by ice in a snowman.
- Screen touch handler (ccToucherEnded in GameLayer.java) is used to add snowflakes and ice to the screen.
- Function to display statistics (showFPS in CCDirector.java).
Icy Rocks Job Scheme
Above is a diagram of the work cycle drawing. It resembles the main cycle, which we considered above, talking about Cocos2D, but our cycle is a bit more complicated. Each time the scheduler timer is triggered, the next frame is calculated and updated. At this time, callback functions and game logic are processed. For example, JBox2D is called to implement game physics. After that, you can start displaying the image.
When the output is complete, a small waiting period is possible - until the FrameTime time comes. This waiting is necessary in order to ensure a stable frame rate. For example, if an application is capable of issuing 100 FPS, it will, thanks to this approach, slow itself down to the required 60 FPS. And the remaining time can be spent on the implementation of more advanced game logic, data preparation and other similar tasks.
Icy Rocks Performance Review
Icy Rocks can be viewed as a single-threaded Android test running on the Android GLTHread stream. During the test, most of the time is spent on executing code from Android Runtime (ART) and working in low-level OpenGL libraries. Some time passes in the machine code of Android, called via the Java Native Interface (JNI) and System.arraycopy.
In the course of the test, in fact, the possibilities of Android Runtime, the graphics capabilities of the Android stack, and the hardware capabilities of the SoC platform are explored.
The Icy Rocks test for Java is demanding for arithmetic logic operations (ALU), the branch prediction subsystem (floating-point comparison), the performance of the associative data translation buffer (DTLB) and the second-level cache (L2).
As already mentioned, the test contains five identical passes. Each of them is divided into 20-second intervals, during which the number of snowflakes and ice floats remains constant. When moving to the next interval, their number increases. Most of these configurations have identical performance characteristics (for example, a similar number of instructions per cycle, the same Java code that is executed most often, and so on). During the test, 32-bit floating point calculations are intensively used - this is the accuracy of collision detection in a physical engine.
What we learned and did thanks to Icy Rocks
During the experiments with Icy Rocks, we found that the physics engine JBox2D uses its own implementation of the methods of the standard library java.lang.Math. It involves huge data structures to support the operation of the Math.sin () and Math.cos () functions. We recommend making changes to the JBox2D code, using the standard java.lang.Math methods for optimal performance.
In addition, it was found that JBox2D joins objects into a pool so that during program execution you do not waste time allocating memory for them and garbage collection. This approach can degrade the performance of the garbage collector, cause heap fragmentation, worsen the locality of references to objects. Our recommendation is to use the standard garbage collector, and not to pool objects. We look forward to further improving the locality of the links and the garbage collector through the use of a memory allocation facility for objects, which is based on the Thread-Local Allocation Buffer (TLAB) in the near future.
Thanks to Android research using Icy Rocks, Intel has contributed to the Android Open Source Project (AOSP) with several enhancements. There have also been many optimizations in the Intel ART code.
Results: it's time for new tests
We opened the code Icy Rocks . This is part of Intel's strategy to change the approach to measuring the performance of Android devices. Existing synthetic performance tests, for the most part, have nothing to do with reality, do not reflect the features of these applications. The data they give out may be distorted even more by optimization. We, for example, recently "improved" in this way CFBench and Quadrant.
We want to convince the Android community to abandon synthetic tests, far from reality and subject to manipulation, and replace them with more believable workloads. Such as Icy Rocks.
Source: https://habr.com/ru/post/279061/
All Articles