Stryker, mutation testing in javascript

Recently, I became acquainted with the method of testing software called “Mutation Testing” and have already become a fan of this approach to writing tests.

First theory

The purpose of mutational testing is to identify ineffective and incomplete tests, that is, this is essentially the testing of tests .

The idea is to modify small random fragments of the source code and observe the reaction of the tests. If after making changes the tests are still passed, then such a set of tests is ineffective or incomplete.

The rule by which the conversion is performed in the source code, for example, substituting true instead of false , is called a mutator (mutational operator) . As mutators, they also use the replacement of signs of arithmetic operations and Boolean operators, zeroing and permutation of variables in places, removal of code branches and others. Changes made to the source code are called mutations . As a result of the acquisition of mutations, the source code mutates and becomes a mutant . After testing, the mutants fall into two categories:

• killed (caught) - those in which deviations were identified and at least one test failed
• survivors (escaped) - those who were able to pass the tests successfully

With automatic mutational testing, a multitude of mutants of the original source code is created, and test sets are run for each of them.

The metrics of the effectiveness of mutation tests is the MSI (Mutation Score Indicator) indicator, which reflects the ratio of killed mutants to survivors. The greater the difference between MSI and the percentage of code coverage of tests, the less informative the criterion for assessing the quality of tests is their percentage of coverage.

It happens that combinations of mutators cause mutually exclusive mutations, and then they say that the resulting mutant is equivalent (to the original program). This is partly why it is incredibly difficult to achieve an MSI of 100% even in small projects.

Now practice

I’ll talk about a framework for automatic mutation testing called Stryker .

To prepare the project, install the stryker-cli package globally:

 npm i -g stryker-cli 

Next, install and save the stryker and stryker-api packages in the project's dev dependencies.

 npm i --save-dev stryker stryker-api 

I’ll use Mocha as a framework for automated testing, and I’m used to using Chai as a library of statements:

 npm i --save-dev chai mocha@3.5.0 

Let's stryker init , this initialization utility will ask a few questions, I chose everything according to my preferences and configuration, plus I added an html item to the list of reports. This is equivalent to this line:

 npm i --save-dev stryker-api stryker-mocha-runner stryker-mocha-framework stryker-html-reporter 

At the end of the configuration, a stryker.conf.js file will be created of the following stryker.conf.js :

 module.exports = function(config) { config.set({ files: [{ pattern: 'src/**/*.js', mutated: true, included: false }, 'test/**/*.js' ], mutate: [], testRunner: 'mocha', testFramework: 'mocha', mutator: 'es5', transpilers: [], reporter: ['html', 'clear-text', 'progress'], coverageAnalysis: 'perTest' }); }; 

We will understand the options and customize it for yourself:

• files - an array of names and name patterns to specify the files needed for testing. As elements you can use:
  
  • string literals, for example, 'src/**/*.js' .
  • InputFileDescriptor objects: { pattern: '', included: true, mutated: false } , where
    
    • pattern is a required field with a name or a name template, but which does not support excluding files through ! unlike string literals. That is, if the file or directory starts with ! and needed in the project, then use this method instead of a string literal.
    • included - an optional field that specifies whether the file should be loaded into the test runner ( true ) or simply copied to the sandbox ( false ). At runtime, you can observe how the .stryker-tmp directory flashed in the project structure, and there are sandboxes with mutants in it, if the project depends on your other module, you must also specify it to copy to the sandbox.
    • mutated - an optional field that specifies whether the file should be subject to mutations.
• mutate is an optional array of names and name patterns to specify the files to mutate. You can do without this array by using InputFileDescriptor objects when selecting files in the files array.
• testRunner - required field, indicates a test runner for tests. Make sure that the appropriate Stryker plugin is installed, for example stryker-karma-runner for using karma as a test runner.
• testFramework - indicates the framework used by tests. By default, uses the value from testRunner
• mutator - optional field, specifies a plugin-set of mutators used in testing, by default es5 .
• transpilers - an optional array field, specifies transpilers that must perform code conversions prior to execution.
• reporter - an optional array field, with which you can choose the format for the presentation of reports after automatic mutation tests.
• maxConcurrentTestRunners is an optional field that defines the number of tests to be performed simultaneously.

As a capacious practical example, I created a project with the following structure

 ├── app.js ├── package.json ├── stryker.conf.js └── test └── app.test.js 

The main file contains and exports only one function.

 // app.js module.exports = { userIsOldEnough: (user) => user.age >= 18 }; 

To substantiate the concept of mutational testing, I will supply the project with 100% coverage with unit tests, even in 2 passes:

 // test/app.test.js const expect = require('chai').expect, app = require('../app'); describe('Site', () => { it('can be visited by an adult', () => { expect(app.userIsOldEnough({ age: 23 })).to.be.true; }); it('can not be visited by a child', () => { expect(app.userIsOldEnough({ age: 13 })).to.be.false; }); }); 

Stryker configuration file looks like this

 // stryker.conf.js module.exports = function(config) { config.set({ files: [{ pattern: 'app.js', mutated: true }, 'test/**/*.js' ], testRunner: 'mocha', reporter: ['html', 'clear-text', 'progress'], testFramework: 'mocha' }); }; 

I also added a couple of scripts to package.json for convenience:

 { "name": "mutations-demo", "version": "1.0.0", "private": true, "scripts": { "test": "istanbul cover _mocha", "posttest": "stryker run" }, "main": "app.js", "devDependencies": { "chai": "^4.1.2", "mocha": "^3.5.0", "istanbul": "^0.4.5", "stryker": "^0.13.0", "stryker-api": "^0.11.0", "stryker-html-reporter": "^0.10.1", "stryker-mocha-framework": "^0.6.1", "stryker-mocha-runner": "^0.9.1" }, "dependencies": { "underscore": "^1.8.3" } } 

Will execute

 npm t 

and now the most interesting begins: you can make sure that all unit tests are passed and they cover 100% of the code

  Site ✓ can be visited by an adult ✓ can not be visited by a child 2 passing (15ms) =============================== Coverage summary =============================== Statements : 100% ( 2/2 ) Branches : 100% ( 0/0 ) Functions : 100% ( 0/0 ) Lines : 100% ( 2/2 ) ================================================================================ 

further, mutation testing begins automatically, and here we get bad news in the form of MSI 50%:

 Mutant survived! Mutator: BinaryOperator - userIsOldEnough: (user) => user.age >= 18 + userIsOldEnough: (user) => user.age > 18 Tests ran: Site can be visited by an adult Site can not be visited by a child Ran 1.50 tests per mutant on average. ----------|---------|----------|-----------|------------|----------|---------| File | % score | # killed | # timeout | # survived | # no cov | # error | ----------|---------|----------|-----------|------------|----------|---------| All files | 50.00 | 1 | 0 | 1 | 0 | 0 | app.js | 50.00 | 1 | 0 | 1 | 0 | 0 | ----------|---------|----------|-----------|------------|----------|---------| 

It follows from the report that the tests are incomplete, since their passage was not affected by the change in the logical operation from >= to > and therefore, they do not check the function in case the website user is 18 years old. This report looks like a diff between commits, but according to the settings, a more beautiful one is generated, in the form of a similar html document.

The repository with this project lies on Github . And so that you can not raise anything and just look at the logs, I added a project to Travis .