support ES6 modules (and support custom compilers/parsers e.g. babel/espree)

Firstly - I love this project! This is going to be hugely useful.

The problem: Esprima will reject ‘import’ statements if the sourceType is not explicitly set to ‘module’:

            case 'import':
                if (state.sourceType !== 'module') {
                    tolerateUnexpectedToken(lookahead, Messages.IllegalImportDeclaration);
                }
                return parseImportDeclaration();

from https://github.com/jquery/esprima/blob/master/src/parser.ts#L1643

This requires Stryker to provide it with sourceType: 'module' as seen here:

var esprimaOptions = {
    comment: true,
    loc: true,
    range: true,
    tokens: true,
    sourceType: 'module' //<=== required for esprima
};

from https://github.com/stryker-mutator/stryker/blob/master/src/utils/parserUtils.ts#L9

Prototype pull req to illustrate: https://github.com/stryker-mutator/stryker/pull/136

Unfortunately, currently there’s no way to tell Stryker this is what you want.

This is the code I am testing:

export default (a, b) => a < b

with this test:

import lessThan from './lessThan.js'
describe('a < b', function(){
  it('it should be true', function(){
    chai.expect(lessThan(1, 2)).to.be.equal(true)
  })
})

and Stryker cleverly realises that

Mutator: ConditionalBoundary
-   export default ((a, b) => a < b);
+   export default ((a, b) => a <= b);

However there’s more - because I’m using ES6 module syntax in my actual mocha tests too, I need to use babel-core/register and some transforms to compile them as they’re required. This works totally fine with Stryker too - I just make the first file it looks at one with babel-core/register, where I have one or two babel plugins beavering away magically transpiling stuff.

require('babel-core/register')({
  "presets": ["es2015"]
})

This should be the end of it - but I’m also a big fan of the new object rest spread syntax: https://github.com/sebmarkbage/ecmascript-rest-spread (https://babeljs.io/docs/plugins/syntax-object-rest-spread/)

I want to use it in the code I’m mutating (not just the tests) in this contrived example:

export default (a, b) => ({ ...{ c: 82, d: 150 } }, a < b)

so I modify my babel-core register hook to use the transform:

require('babel-core/register')({
  "presets": ["es2015"],
  "plugins": [
    "transform-object-rest-spread"
  ]
})

But it doesn’t work - because Stryker loads the files it is going to mutate into Esprima, it doesn’t require them directly so Babel can’t transpile it.

To fix this, I need to go into: https://github.com/stryker-mutator/stryker/blob/master/src/MutatorOrchestrator.ts#L45 and add babel to transform the file:

                var fileContent = fileUtils.readFile(sourceFile);
                fileContent = babel.transform(fileContent, { plugins: ["transform-object-rest-spread", "transform-runtime"]}).code;
                _this.reportFileRead(sourceFile, fileContent);

(I tried doing this later (just before Esprima parses it) but it caused weird issues where Stryker wouldn’t generate any mutations, or claim that all mutations passed - I think probably because it meant reportFileRead and other functions had differing opinions about the state of the source code.)

We need both these transforms because with just the first, it adds a lot of extra code into the file which gets Stryker over excited:

Mutant survived!
Mutator: UnaryOperator
-   var _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; };
+   var _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i--) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; };

Tests ran:
    a < b it should be true

14 mutants tested.
0 mutants untested.

and it generates over 14 mutants, all mostly meaningless variations on Babel’s polyfill. “transform-runtime” extracts the polyfill out into a separate file so Stryker only sees a single function call and isn’t tempted to mutate anything.

With Babel:

Mutator: ConditionalBoundary
-   export default ((a, b) => (_extends({ c: 82, d: 150 }), a < b));
+   export default ((a, b) => (_extends({ c: 82, d: 150 }), a <= b));

Tests ran:
    a < b it should be true

2 mutants tested.
0 mutants untested.
0 mutants timed out.
1 mutants killed.
Mutation score based on covered code: 50.00%
Mutation score based on all code: 50.00%

That’s more like it!

Prototype pull req to illustrate: https://github.com/stryker-mutator/stryker/pull/134

There is an alternative to using Babel, which is to use Espree instead of Esprima: https://github.com/eslint/espree Espree is based on Esprima but has support for more experimental features (among them object rest spread).

It was a pretty simple swap - in https://github.com/stryker-mutator/stryker/blob/master/src/utils/parserUtils.ts#L30 I just swap out esprima for espree and add

    ecmaFeatures: {
      experimentalObjectRestSpread: true
    }

as an option and it just works.

With Espree:

Mutator: ConditionalBoundary
-   export default (a, b) => ({ ...{ c: 82, d: 150 } }, a < b)
+   export default (a, b) => ({ ...{ c: 82, d: 150 } }, a <= b)

Tests ran:
    a < b it should be true

2 mutants tested.
0 mutants untested.
0 mutants timed out.
1 mutants killed.
Mutation score based on covered code: 50.00%
Mutation score based on all code: 50.00%

Prototype pull req to illustrate: https://github.com/stryker-mutator/stryker/pull/135

As you can see, Espree is not transforming the code, just correctly parsing it, so the mutation snippet is a bit truer to life. Babel is a lot more powerful in the sense that you can transform pretty much anything you like into anything else, but you’d need Stryker to understand sourcemaps in order to provide perfect feedback, and you’d need to make sure any Babel transform left only the minimum amount of ‘mutative noise’ left over (or otherwise wrap any mutative noise in comments that Stryker could ignore - sounds messy).

So there are several options here - hacking it in like in my pull reqs is not ideal, so perhaps there needs to be a way to expose a callback to the config whenever a file is read or parsed so, at the very least, people can provide their own compilers. Alternatively or alongside which, should it be possible to provide compilers as Stryker plugins?

What do you guys think? Is this kind of configurability on the roadmap, and if so do you mind if I take a crack at it?

Apologies - I didn’t use the correct commit style (https://github.com/stryker-mutator/stryker/blob/master/CONTRIBUTING.md) on the pull reqs - they’re just prototypes to illustrate the issue, I don’t expect they’ll be merged!

Thanks, Will

[simondel]

With stryker-javascript-mutator and stryker-babel-transpiler we have support for ES6 modules. React support will be added with #525.

[CrazyBS]

I am extra excited to see this working in a babel/webpack pipeline. I am using es6 and react plugins in babel and have no luck getting version 0.6.0 working with the mutations turned on. It does run the dry-tests successfully…

I recognize that there is significant challenges to fully parsing and mutating the raw source code no matter what stage-? features the user is adding into their babel plugins. It would be great if this plugin didn’t care at all and just registered to the output of babel and mutated the compiled code with coverage to help guide it (is that a thing?).

You have your work cut out for you and I if this feature is successful, it will be the standard include in every client project I do. Keep up the good work!