About speed testing or how not to write tests

Recently I saw a post Test browser performance with HTML5 Canvas . In the results, IE9 began to show crazy digits - 350+ fps.

This, of course, is good, but for some reason, browsers, which in other tests of javascript and canvas showed no less performance, in this test showed several times (and sometimes dozens of times) lower fps (subject to running on Windows). platform, but more on that later).

Under the cut, I'll show you why the test shows the speed of the HTML5-Canvas, and at the very end there will be a screen with 470 fps for FF4 without any photoshop, for a start we will analyze what is wrong in this test.
')

Well, for starters, let's see what 200 fps is. This is 200 draws per second. Those. a drawing event should occur approximately every 5 ms. If you look at the test code, then there really is a line:

setInterval(moveIt,1); 

Indeed, the “moveIt” should be executed every 1ms judging by the specification ( WhatWG , W3C ). But there is no such thing as the minimum time after which the handler should be called.

And then the peculiarities of the implementation of the event queue in browsers come into force. There is such an interesting test that should show the minimum accuracy of setTimeout. I had 4ms in Chrome, but in FF4 it was clearly kept for 10ms. Perhaps this is due to the granularity of working with a timer for Windows systems (without using high-performance counter or multimedia extensions) (at the end there is a sign with the results).

So, as the goal of this topic is not to discuss the specifics of the implementation of setTimeout, but to discuss how you can really measure the performance of a particular operation. As we have seen, setTimeout is not suitable (maybe only “not yet suitable”?) - it does not allow the browser to fully load with any task.

You can’t do it without a timeout either (i.e., just carry out an operation in a loop), because the user will not see anything - you must give the browser time to display the result of the operation.

A possible solution to this problem was published by Davin Baron : use window.postMessage ( WhatWG , W3C , MSDN ). A code that implements the setTimeout analog, but with a minimum (preferably zero) delay (I modified David’s code by adding an analog to setInterval (correct me if fn.apply is needed there):

 // Only add setZeroTimeout to the window object, and hide everything // else in a closure. (function() { var timeouts = []; var messageName = "zero-timeout-message"; // Like setTimeout, but only takes a function argument. There's // no time argument (always zero) and no arguments (you have to // use a closure). function setZeroTimeout(fn) { timeouts.push(fn); window.postMessage(messageName, "*"); } function intervalHelper(fn) { var h = function() { fn(); setZeroTimeout(h); }; return h; } function setZeroInterval(fn) { setZeroTimeout(intervalHelper(fn)); } function handleMessage(event) { if (event.source == window && event.data == messageName) { event.stopPropagation(); if (timeouts.length> 0) { var fn = timeouts.shift(); fn(); } } } window.addEventListener("message", handleMessage, true); // Add the one thing we want added to the window object. window.setZeroTimeout = setZeroTimeout; window.setZeroInterval = setZeroInterval; })(); 

The difference is huge ( demo ):

 100 iterations of setZeroTimeout took 19 milliseconds. 100 iterations of setTimeout(0) took 393 milliseconds. 

With this minimal change, we have a screenshot in the header in the case of Firefox4 and Direct2D / DirectWrite enabled (you can see it in about: support )

Here is the HTML5 Canvas “test” type with the change applied.

A few words about Google Chrome: under non-Windows platforms, it often shows 200+ fps in this test on even not very strong machines (provided that hardware acceleration is enabled). On Windows, it rests on the VSync limit - 60 fps. There is an issue about this, for which you can vote by putting an asterisk.

Well, do not forget to watch about: flags - there is a Accelerated Compositing / GPU Accelerated Canvas 2D GPU.

Conclusion

When writing a test / benchmark, if a browser has shown an inexplicably big difference, do not begin to mistakenly think that this browser is mega-super optimized. It is quite possible that the test simply tests not what it seems. For example, a test that, in theory, should show the speed of HTML5-Canvas actually showed the features of the implementation of setTimeout and enable / disable VSync.

Pps. To show the curvature of this test completely, open the test on FF4 with my patch and tab on the other tab (almost immediately), then in 15-20 seconds go back - I had about 400 fps and slowly fell down. Proof

In general, before publishing a test, check whether it tests well and if it tests something sensible, rather than just drawing a beautiful picture with digits.

SetTimeout Minimum Delay

Having collected the comments, I made the following table of test results:

Opera 11 (Win)	2
Chrome 10-11 (Win)	four
Firefox (Win)	ten
Safari, OS X 10.6.6	ten
iphone	ten
IE 8	15.6

As comments arrive, I will replenish.

On this occasion, it is worth noting that, as recommended by HTML5 from WhatWG / W3C, setTimeout / setInterval should work like this:

The setInterval () method must run the following steps:
5. If timeout is less than 10, then increase timeout to 10

The setTimeout () method must run the following steps:
5. If this is the current time, the timeout is less than 4, then increase the timeout to 4.

Those. less than 4 ms for setTimeout and 10 for setInterval should not be. That shows an obvious error in any browser, where the “original” test gives more than 100 fps (as it uses exactly setInterval). 250 fps is the maximum when using setTimeout.

UPDATE:
I made a copy of the test in which I increased the number of polygons (in the field up to 72 segments and letters 30 times).
Now I have the results (Windows7 SP1 x64, nVidia gtx 260, Core Quad 2.4ghz):

Chrome 12.0.703.0 canary build	33 fps
IE 9.0.8112.16421	26 fps
Firefox 4.0	58 fps

No sharpening, etc. I did not do it (except for setZeroTimeout, which also increases the speed of the original “test” in IE9). That mega-huge difference as in the original test, I do not see.

As a result, I will say one thing: a crookedly written test can “embellish” the result dozens or even hundreds of times.