Improve karma to not always run the whole test suite

Hi there,

my goal is to improve karma to not always run the whole test suite when a file changes. To avoid lots of effort going in the wrong direction I would like to talk about this first, before starting the implementation.

Ideal Goal

Improve karma to not always run the whole test suite when a file changes

Motivation

This is an important issue in larger projects with a lot of tests. Changing a file (using autoWatch or grunt-karma) usually triggers a run of the whole test suite. This is not always necessary and can be avoided.

There are two isolated sets of files that usually trigger a re-run of all the tests:

1. source files
2. spec files

Regarding the first category one cannot do much easily. What one can do, is to derive the set of spec files four each source file from the first test run (maybe Istanbul can help here?). This is not what my proposal is about (although it should be done, but not as a first step).

What I want to improve is the second set of files: when changing a spec file, karma should run only this particular spec file

So let’s talk about the …

… Realistic Goal

Improve karma to not run the whole test suite when only a spec file changes

Proposed changes (i.e. backwards-compatible additions)

1. Add a sixth configuration option for the files pattern object (current keys are pattern, watched, included, served, nocache). It could be named isspec and defaulting to false. If set to true for file A, karma could (when the watcher detects a change in A) load (i.e. include as in the included setting) only those files who have isspec == false plus A. Conversely, if the watcher detects a change in file B which we assume to have isspec == false, then it would just load the files as usual.
2. Change No. 1 only works when running karma standalone with autoWatch option. It does not work in conjunction with grunt-karma (e.g.). Here, we can have a karma server started in background and tests are only invoked when the build-system detects a change (grunt-karma then uses the runner api to invoke a test run) **. The source code reveals: the whole config object is passed to the runner. But the runner does not respect all of its properties. The easiest solution would be to allow for a complete reconfiguration of config.files (just using what is passed to the runner). However, this might affect performance, because if a developer switches between source and spec files (making changes), karma would have to re-do all the globbing again and again. I have the feeling that additional care has to be taken to ensure that this performance loss does not outweigh the benefits of switching between different config.files configurations.

** The ideal setup I would like to support involves using (1) grunt-watch, setting up two different watch-targets, one for watching the source files and one for watching the spec files, and using (2) grunt-newer for the spec-files watch task to be able to dynamically pass only the changed files to the grunt-files array, e.g. like so: ‘newer:karma:run’. Respecting the grunt-files array was only “recently” (in January) made possible.

What do you think?

What do you think? If anything is unclear, I’m happy to try and explain more thoroughly. I’m feeling that especially change No. 2 could be solved differently in many ways (e.g. letting grunt-karma do more of the work, like WebStorm does, for which the changes in runner.js were introduced in the first place, i.e. the changes to allow for notification on added/removed/changed files).

Regards, Nicolas

[stefaneidelloth]

I also first commented with “+1” but then I learned that this will spam all persons that already wrote a comment. => Please use the thumbs up feature on the issue description.

[AndrewEastwood]

+1