RoboHornet: a new approach to browser performance testing

Synthetic performance tests cause a lot of censure - they measure wrong and wrong, and the test result often has little to do with real speed. Or even worse - the test turns into a marketing tool and consistently shows the best result in any one browser. RoboHornet is conceived as a test devoid of these shortcomings. Created by Google, it was published on Github on September 24 - its development will be carried out openly. The results calculation system is also based on crowdsourcing. The developer community determines which tests to include in RoboHornet and how much weight each test will have by voting.

RoboHornet tests the performance of JavaScript, DOM, Canvas, SVG, Local Storage and even animated GIFs. The community of testers is coordinated by the so-called " stewards ", among whom are employees of Google, Facebook, Sencha, active participants in projects YUI, HTML5 Boilerplate, jsPerf, jQuery and others. The idea behind RoboHornet is very simple: any popular test forces manufacturers to strive to ensure that their browser shows the best results in it. If you create and make popular a test that covers all important aspects of the work of web applications, and not just JavaScript, or, for example, Canvas, this will contribute to a more balanced development of browsers.

Each member of the community focuses on the problems that he faces in his application, and seeks to write more tests in this area, to raise their weight in the overall assessment. With the joint efforts of many programmers, the test suite over time becomes more complete and versatile.

RoboHornet is still in the early alpha stage, but it started pretty confidently - among the stewards are such reputable people as Paul Irish and Yehuda Katz. Tom's Hardware, which conducts regular “races” of the Web Browser Grand Prix browsers, has already run out new tests for all major browsers on Windows 7 and 8, Ubuntu 12.04.1, OS X Mountain Lion and Android 4.1.1
')
Not without criticism. Despite the fact that under Windows 8, the tenth Internet Explorer showed an excellent result, confidently overtaking all competitors, Microsoft said it considers this test to be completely useless and far from the real world. According to Internet Explorer Marketing Director Roger Kapriotti, who published a critical post on the IE developer blog on September 25, the benefits of IE10 are even higher in real-world conditions.

To prove his words, he brought a demo with the modest name RoboHornet Pro , in which the tests included in RoboHornet were launched in parallel with the work of rich web page animation. At the same time, Google Chrome hung up and stopped responding to user actions, while IE10 remained responsive and cursed the animation smoothly. What prevents Microsoft developers from participating in the creation of correct, valid tests as part of the RoboHornet community, and not on a separate page of the corporate website, Capriotti did not specify.

Since the composition, content and weight of tests in different releases of RoboHornet will continuously change, the so-called RoboHornet index is selected as the main assessment. 100 points correspond to the average result of the totality of all current stable versions of popular browsers on the reference hardware. The benchmark in the current version is a MacBook Pro with the following characteristics:

• Quad Core 2.2GHz Intel Core i7
• 500GB 5400-rpm HDD
• AMD Radeon HD 6750M graphics card with 512MB GDDR5 memory
• 4GB ROM
• Screen resolution 1440 by 900 pixels
• OS: dual-boot Windows 7 Service Pack 1 + Mac OS X 10.8.1

The reference iron will be updated approximately once a year. Why such an evaluation scheme was chosen is described in more detail in this document .

Now on the official site of RoboHornet.org you can pass as a complete set of tests (each test is cursed so many times to eliminate the influence of random factors, so on the average gland the whole process may well take 5 - 10 minutes), and the subsets chosen like this to match typical requests of certain web applications and frameworks - Google Maps, YUI, jQuery and others. You can vote for important, in your opinion, tests or offer your own in the project repository on Gitkhab .