PhantomJS + JSCoverage + QUnit or JS console unit tests with coverage calculation

Let's talk about the case when you need to automate the launch of tests and the collection of statistics of coverage, for example, for a hypothetical client JS library. The task is not entirely trivial, since a full-fledged browser is required for the library to work properly - the library is a visual wrapper over standard form components. The library should be written so that all interaction with its objects can be performed using the methods they provide, and not only through direct manipulations with the DOM (i.e., any user action can be triggered not only by an event, say, click something, but also by hand through the method). But nevertheless, it is necessary to have this DOM, so that the results of the work of methods in addition to changing the internal state of objects are also displayed in the DOM. In general, like the application on Sencha (ExtJS).

To achieve these goals, you need a controlled browser, a framework for running tests and a utility that allows you to calculate code coverage with tests, as well as some code that connects all the components.

Concept

One of the solutions is JSTestDriver , which has already been written about here, but in the case under consideration this option did not fit, since the entire solution should be launched as a console, should not produce unnecessary services, or browser windows, should be started and die immediately upon completion. Plus, at the moment JSTestDriver does not know how to exclude connected files from the Coverage Report, which greatly distorts the picture, because Considering coverage, for example, for connected jQuery or MooTools is superfluous. On Habré was an overview of the simple launch of QUnit tests using PhantomJS , but without counting the coverage or assembling the page (which is necessary just for coverage).
')
Taking into account the specified requirements, the solution was born based on the PhantomJS + JSCoverage + QUnit bundle , which I posted on Google Code. Let us dwell on the components:

• PhantomJS - console (headless) browser controlled via JS API;
• JSCoverage is a console parser for JS files, injects an increment in them for each executable line. array element to count the number of executions. There are minor limitations - only strings are involved that can be executed at all, i.e. no comments, for example. For the notation of objects and arrays, only the line is considered, where the notation started;
• QUnit - a small JS framework for convenient and quick creation of unit tests, including asynchronous, runs in the browser as a page with connected scripts.

Launch

For the utility to work correctly, you need to download and unpack two archives with executable files PhantomJS (dynamic) and JSCoverage , the paths to them can be changed in batch files, and you can specify the correct name of the test group for execution there. In order to tie together all the components, the auxiliary functionality in question was written. To set up a set of tests and scripts for inclusion in the triggered page, one configuration file is used:

var config = { includes: [ // ,       { file: 'lib/jquery-1.7.1.js', coverage: false //          }, { file: 'lib/json2.js', coverage: false }, { file: 'lib/testable.js', coverage: true //     —  } ], testCases: [ //    ,   ,    ,    { location: '/tests', pattern: /.+Test\.js/g, recursive: true } ], target: { //     location: '/target' } }; 

After the launch of the “Most Main Batch file” (in the example from the repository run-tests.cmd or run-tests2.cmd , depending on which set you plan to run), the following series of actions take place:

1. Collect all js files in one folder for processing by JSCoverage;
2. Go through them JSCoverage'om;
3. Collect links to other included files that are not involved in the calculation of coverage (libraries, QUnit itself, auxiliary components for testing, etc.);
4. Find all files with test cases;
5. To assemble a page from the elements described above for execution in the normal QUnit order;
6. Launch this page with scripts for execution from under PhantomJS;
7. Pick up the QUnit report and execution statistics - coverage, number of completed, successful and fallen tests from the “juraser's window”;
8. Generate a human-readable report from all of this;
9. Add to the folder reports, as well as everything you need to re-launch.

Reports

After doing the manipulations in the folder with the utility will be a subfolder / target /, which will be folded:

• test.html is the same page with the necessary list of scripts to run in webkit-like browsers (PhantomJS restriction, it cannot be referenced with the file: // protocol yet), allows you to use all your favorite dev tools to debug test curves in a familiar visual environment (the order is this - we run the console test, we look in the browser, we fix it, we run it again, we look at the updated version in the browser);
• test-result.html - test execution report, collected coverage statistics, without directly performing tests:
  
• * .js.html - coverage reports for each file included in the report:
  
• coverage.json - raw statistics on the number of lines filled by files, just to see in any case what JSCoverage collected.

Total

+ Collect statistics coverage;
+ Automatic assembly of resources for testing, the user is required to configure once and simply run the batch file;
+ Run from under the console;
+ Standalone (to start it does not need anything, except for two programs that do not need to be configured or installed - just unpack the archives);
+ Visual report after execution;
- Testing takes place in the conditions of the webkit browser, which does not allow to check, for example, the features of browsers with arrays;
- The collector currently works only with relative paths, in the near future there will be support for absolute paths and links;
- There is no integration with Maven or something else for continuous integration - in the near future.

At the moment, the utility can be regarded as proof of concept, which copes well with its task, simplifies life, but is not without drawbacks. In any case, I plan to develop functionality as needed and as far as possible. The kit also includes a small utility for running tests from JSTestDriver within the framework described above, so you can drive tests to the IDE in PhantomJS, but at the moment it does not know how to run asynchronous tests.