Support mutating TypeScript code

Stryker is only capable of mutating JavaScript. For TypeScript projects this results in an unprecise mutation score as generated code is being mutated and because some mutations would normally cause compilation errors.

So as an extension of issue #343, the `stryker-typescript plugin will be developed. @thomasp24 has already done a great deal for this and the current code can be viewed on the support-typescript branch. Based on that @nicojs created a typescript-mutation-poc in various features are demonstrated needed for the changes here.

Todo list

We’re working on this in the stryker-typescript branch.

• Gather info on projects that want this plugin so we can test it.
  • Stryker itself
  • Typed html: https://github.com/nicojs/typed-html
  • Link parent bin: https://github.com/nicojs/node-link-parent-bin/
  • Install local: https://github.com/nicojs/node-install-local
  • Moneyboom ch frontend: https://github.com/wearesho-team/moneyboom-ch-frontend (see comment 327761543)
  • Wearesho-site https://github.com/wearesho-team/wearesho-site (see comment 327761543)
• Update stryker-api for a Transpiler api
• Update stryker-api’s Mutator plugin. Update stryker’s Mutator for this structure.
• Update stryker to use the Transpiler api.
• Create TypescriptConfigEditor responsible for enhancing stryker.conf.js with typescript’s config.
• Create TypescriptTranspiler responsible for transpiling ts code and generating sourcemaps
• Write unit tests for TypescriptTranspiler
• Create TypescriptMutator responsible for mutating typescript code and validating that it can still compile.
• Write unit tests for TypescriptMutator.
• Add Typescript mutator equivalents for every js mutator in stryker.
  • ArrayDeclaratorMutator
  • BinaryOperatorMutator
  • BlockStatementMutator
  • BooleanSubstitutionMutator
  • LogicalOperatorMutator
  • RemoveConditionalsMutator
  • UnaryOperatorMutator
  • UpdateOperatorMutator
• Refactoring in Stryker:
  • Refactor InputFileResolver to now also read in the input files initialially
  • Make InputFileResolver responsible for reporting OnSourceFileRead and onAllSourceFilesRead
  • Refactor current Mutators in Stryker to be implemented using the new MutantGenerator api
  • Refactor - check if code coverage analysis still works. Stryker is now calculating its own line/column offset based on position in a file. It’s doing that 0-based, so the first line is 0 and first column is 0. By istanbul inspected code is first line 1 and first column 0.
• Notify users if they specify a transpiler and a coverage analysis level, because this is not yet supported.
• Add unit tests for the ChildProcessProxy
• Don’t start a transpiler ChildProcessProxy if there is no transpiler
• Rename mutantGenerator to mutator everywhere (see comment 328269389)
• Add ‘transpiled’ property to the FileDescriptor so users can select if a file should be transpiled. Defaults to ‘true’.

Way it will work

When running Stryker on a TypeScript project, we won’t mutate your javascript anymore. Please read the blog article about the Stryker 1.0 roadmap to know why. Instead, we’ll mutate your TypeScript code. We split the responsibility of TypeScript mutation testing in 2 blocks:

1. Mutate typescript code.

The mutating will be done basically the same way as it is now for javascript code. We’ll walk the AST and create mutations on copies of the typescript nodes. This will keep the process really fast. It is OK for these mutations to cause compilation errors as they’ll be caught during transpiling of the typescript code.

1. Transpile typescript code.

All code will be transpiled for the initial test run, and then for each mutant after that. This has some advantages over other ways of working:

• By first generating all mutants, we can later decide to devide the transpiling workload over multiple processes (however, we’ll start with using the host process for this PR)
• By running the transpiler for each mutant, we can mutate our typescript code as complex or simple as we want. We don’t have to worry about the javascript code it will produce. This contrary to transpiling once and than go find-and-replacing bits of javascript code (using source maps). That might be faster, but we would be limited in what we can mutate, as complex typescript constructs (like decorators for example) can result in complex javascript, different for each target (es5, es2015, es2016, …).

There is some added complexity though: we want to use coverage analysis on transpiled code too. This means that we need to actively keep track of source maps if we want to know which line, column in the source code is hit by a particular javascript statement. Fortunately, TypeScript has excellent source map support. We might skip coverage analysis for this PR. Lets make it work first.

The transpile api will be added to accommodate this process. You’ll be able to configure multiple transpiler plugins. Each transpiler gets the output of the previous transpiler. This makes it possible to first transpile using TypeScript and next transpile that output using a bundler like web pack. Each transpiler is responsible for its own source maps.

A picture to illustrate:

foo.ts   ==> Typescript  ==> foo.js ==> Webpack ==> foobar.js
bar.ts   ==> Transpiler  ==> bar.js ==> Transpiler

So foo.ts:3:5 might map to foo.js:5:6 which maps to foobar.js:52:46.

Pretty complex stuff, but makes it really pluggable.

In order to keep things as fast as possible, transpilers know exactly which file is changed for a mutant. This makes it possible to use the TypeScript “watch source files” feature to not do too much work. It goes without saying that we want to do as much work as we can in-memory.

Usage

Your future stryker.conf.js file might look like this:

// stryker.conf.js
module.exports = functions(config) {
  config.set({
    tsconfigFile: 'tsconfig.json',
    mutate: 'src/**/*.ts',
    mutatorGenerator: 'typescript',
    transpilers: [
      'typescript'
    ]
  });
}

Do you want to use this on your project? Please let us know if we can access the source code for testing purposes!

Note: This post will be updated to always show the latest info.

[omar10594]

I have a private project where we use typescript and webpack (I will make a new public small project with similar configuration).

For test with stryker, we first build with webpack, and then use stryker with the js code.

[Horat1us]

I have two open-source projects, where I want to test this

https://github.com/wearesho-team/moneyboom-ch-frontend https://github.com/wearesho-team/wearesho-site