JavaScript bulletproof tests

Writing speed tests JS is not as easy as it seems. Without even touching on cross-browser compatibility issues, one can fall into many pitfalls.



That is why I did jsPerf . A simple web interface so that everyone can create and share tests, and check the speed of various code fragments. You don’t need to worry about anything - just enter the code whose performance you need to measure, and jsPerf will create for you a new testing task that you can then run on different devices and in different browsers.



Behind the scenes, jsPerf first used the library on JSLitmus , which I called Benchmark.js . Over time, it overgrown with new opportunities, and recently John-David Dalton rewrote everything from scratch.



This article sheds light on various tricky situations that may occur during the development of JS tests.

')

Test Templates

There are several ways to run a test part of the JS code to test for speed. The most common option is template A :

var totalTime, start = new Date, iterations = 6; while (iterations--) { //     } // totalTime →  ,      totalTime = new Date - start; 

The code under test is placed in a loop that is executed a specified number of times (6). After this, the start date is subtracted from the end date. Such a template is used for testing frameworks SlickSpeed , Taskspeed , SunSpider and Kraken .

Problems

With a constant increase in the speed of devices and browsers, tests using a fixed number of repetitions, more and more often give 0 ms as a result of the work that we do not need.

Pattern B

The second approach is to calculate how many operations are performed in a fixed time. Plus: no need to choose the number of iterations.

 var hz, period, startTime = new Date, runs = 0; do { //     runs++; totalTime = new Date - startTime; } while (totalTime < 1000); //  ms   totalTime /= 1000; // period →      period = totalTime / runs; // hz →     hz = 1 / period; //     // hz = (runs * 1000) / totalTime; 

Runs the code in about a second, i.e. until totalTime exceeds 1000 ms.



Pattern B is used in Dromaeo and V8 Benchmark Suite .

Problems

Due to garbage collection, engine optimizations and other background processes, the execution time of the same code may vary. Therefore, it is advisable to run the test many times and average the results. V8 Suite runs tests only once. Dromaeo - five times, but sometimes this is not enough. For example, to reduce the minimum test time from 1000 to 50 ms, so that there is more time left for repeated runs.

Template C

JSLitmus combines two templates. He uses pattern A to run the test in a loop n times, but the cycles adapt and increase n at run time until the minimum test run time is reached — that is, as in pattern B.

Problems

JSLitmus avoids the problems of Template A, but does not escape from the problems of Template B. For calibration, 3 fastest test repetitions are selected, which are subtracted from the rest. Unfortunately, the “best of the three” is not statistically the best method. Even if you run the tests many times and subtract the calibration average from the average result, the increased error of the result obtained will eat the entire calibration.

Pattern D

Problems of the previous patterns can be eliminated through the compilation of functions and loop development.

 function test() { x == y; } while (iterations--) { test(); } // …  → var hz, startTime = new Date; x == y; x == y; x == y; x == y; x == y; // … hz = (runs * 1000) / (new Date - startTime); 

Problems

But there are drawbacks here too. Compiling functions increases memory usage and slows down. If you repeat the test several million times, you create a very long line and compile a giant function.



Another problem with the development of the cycle is that the test can arrange exit through return at the beginning of work. It makes no sense to compile a million lines, if the function returns to the third line. It is necessary to track these moments and use the template A in such cases.

Removing body function

Benchmark.js uses a different technology. We can say that it includes the best sides of all these templates. We do not deploy cycles to save memory. To reduce the factors affecting accuracy, and to allow tests to work with local methods and variables, we extract the function body for each test. For example:

 var x = 1, y = '1'; function test() { x == y; } while (iterations--) { test(); } // …  → var x = 1, y = '1'; while (iterations--) { x == y; } 

After that, we run the extracted code in a while loop (pattern A), repeat until the specified time passes (pattern B), repeat everything together so many times to get statistically significant results.

What you need to pay attention

The timer doesn't quite work correctly.

In some combinations of OS and browser, timers may not work correctly for different reasons . For example, when booting up Windows XP, the interruption time is usually 10-15 ms. That is, every 10 ms the OS receives an interrupt from the system timer. Some older browsers (IE, Firefox 2) rely on the OS timer, that is, for example, a call to Date (). GetTime () receives data directly from the OS. And if the timer is updated only every 10-15 ms, this leads to an accumulation of measurement inaccuracies.



However, this can be circumvented. In JS, you can get the minimum unit of time . After that, you need to calculate the test run time so that the error is no more than 1%. To obtain an error, you need to divide this minimum unit in half. For example, we use IE6 on Windows XP and the minimum unit is 15 ms. The error is 15 ms / 2 = 7.5 ms. To make this error no more than 1% of the measurement time, divide it by 0.01: 7.5 / 0.01 = 750 ms.

Other timers

When run with the --enable-benchmarking flag option, Chrome and Chromium give access to the chrome.Interval method, which allows you to use a high-resolution timer down to microseconds. When working on Benchmark.js, John-David Dalton met a Java-second timer in Java , and accessed it from JS via a small java-applet .



Using a high-resolution timer, you can set a shorter test time, which results in fewer errors as a result.

Firebug disables JIT in Firefox

The launched Firebug add-on disables the built -in just-in-time compilation , so all tests are performed in the interpreter. They will work there much more slowly than usual. Do not forget to disable Firebug before tests.



The same, though to a lesser extent, applies to Web Inspector and Opera's Dragonfly. Close them before running the tests so that they do not affect the results.

Browser features and bugs

Tests using loops are subject to various browser bugs — an example was demonstrated in IE9 with its dead-code removal feature. The bugs in the Mozilla TraceMonkey engine or caching the querySelectorAll results in Opera 11 can also interfere with getting the right results. Need to keep them in mind.

Statistical significance

The article by John Resig describes why most tests fail to produce statistically significant results. In short, you should always evaluate the magnitude of the error of each result and reduce it in all possible ways.

Cross-browser testing

Test scripts on real different browser versions. Do not rely, for example, on compatibility modes in IE. Also, IE up to version 8 limited the script to 5 million instructions. If your system is fast, then the script can complete them in half a second. In this case, you will receive a “Script Warning” message in the browser. Then you have to edit the number of allowed operations in the registry. Or use the program that fixes this limitation. Fortunately, IE9 has already removed it.



Conclusion



Whether you run multiple tests, write your test suite, or even a library — there are a lot of hidden points about JS testing. Benchmark.js and jsPerf are updated weekly , fix bugs and add new features, increasing the accuracy of the tests.