Support mutation testing with angular cli

We’re investigating the possibility of supporting angular in Stryker, a mutation testing framework. I was wondering if it was possible to run test with ng test --watch=false --single-run=false and keep open the karma server so we can fire karma run commands.

The idea would be that we make small code changes and rerun the tests with karma run. Ideally we would be able to distinguish between a failed test and a compile error.

[filipesilva]

Heya @nicojs, I think that would be a nice addition, yes. We’ve recently added another binary as an API too in https://github.com/angular/angular-cli/pull/12022. I’d ask you to use roughly the same approach as https://github.com/angular/angular-cli/blob/master/packages/angular_devkit/benchmark/src/main.ts uses.

Also, you might be interested in https://github.com/karma-runner/karma/pull/3153. We’d like that functionality in so that we can test Karma rebuilds, and I think it might be relevant for you too since there’s a lot of tooling involved in Stryker.

[nicojs]

Hi again 👋

I’ve got some good news. I’ve decided on another approach, implemented it and released it. It works 👍

However, it relies on a private api: '@angular/cli/lib/cli'. Before I close this issue, I would like to make a PR to make the cli public. It would allow programmatic execution of commands:

import { cli } from '@angular/cli';
cli({
    cliArgs: ['test', '--progress=false', /* ...etc */],
    inputStream: process.stdin,
    outputStream: process.stdout
});

@filipesilva would that be OK?

To use angular with mutation testing, you can use this guide: https://github.com/stryker-mutator/stryker-handbook/blob/master/stryker/configuration/angular.md

Basically it works like this:

• It starts the ng test command and keeps the server open.
• It mutates files one by one and reports the result

The implementation is based on the intellij karma plugin, which also executes ng test and keeps the server open. Nice find!

Thanks for making this possible by merging #11181