Total rebuild test time in medium and large projects

Versions

    _                      _                 ____ _     ___
   / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
  / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
 / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
/_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
               |___/
@angular/cli: 1.4.7
node: 8.8.0
os: darwin x64
@angular/animations: 4.4.5
@angular/cdk: 2.0.0-beta.12
@angular/common: 4.4.5
@angular/compiler: 4.4.5
@angular/core: 4.4.5
@angular/flex-layout: 2.0.0-beta.9
@angular/forms: 4.4.5
@angular/http: 4.4.5
@angular/material: 2.0.0-beta.12
@angular/platform-browser: 4.4.5
@angular/platform-browser-dynamic: 4.4.5
@angular/router: 4.4.5
@angular/cli: 1.4.7
@angular/compiler-cli: 4.4.5
typescript: 2.3.4

Node v8.8.0
NPM v5.4.2
Mac OS High Sierra

Observed behavior

Running tests on medium and large projects take a long time to start, recompile and rerun tests. We have are continually increasing the timeouts within the karma.conf file for the amount of time given to Chrome before it disconnects from no activity (increased both captureTimeout and browserNoActivityTimeout).

We have also experienced issues where, due to the timeout, we have multiple (headless) Chrome instances running the tests where Karma has started another instance (although this sounds like an issue with Karma, but I thought I’d mention it)

On our project in particular, the vendor bundle is ~15mb, which is taking a long time to load when running the test in debug mode (watching the network activity in Chrome before the tests start).

Opening this issue for a broader discussion with @filipesilva

[mattem]

As an update…

We are in the process of moving our code and all its libraries into a mono-repo, as part of that process I’ve updated to Angular 5.1.0 and CLI 1.6.1. We are now running ~ 3000 specs. The code splitting using the lazy chunks have definitely helped the build / rebuild cycle, and loading a ‘suite’ of tests into the browser is much faster than loading the one bundle.

However, we am still hitting issues with the time taken to actually run the tests. A full run takes about 20 mins when it completes. The main culprit seems to be our Form based tests. (a bit of background: we have user definable forms on the server side, any component could appear in a generated form, meaning the whole form has to be dynamic and injected at runtime). Compiling the module that contains all these form components (100+) takes a long time, and as this is done per test… you can see where this is going. This seems okay when running in isolation (fit), but when running with the rest of the tests, it seems to slow down dramatically over time, even on seemingly unrelated tests, eventually causing Karma to disconnect and crash - on first glance it seems like a memory leak somewhere, but I have not done any investigation yet.

This is generally what we get on CI, with random timeouts when running locally (note that I’ve disabled a lot of tests still, hence the lower numbers)

HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) SUCCESS (0 secs / 6 mins 50.281 secs)
20 12 2017 15:12:03.779:WARN [HeadlessChrome 0.0.0 (Linux 0.0.0)]: Disconnected (1 times)
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnectedundefined
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) DISCONNECTED (8 mins 17.695 secs / 6 mins 50.281 secs)
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnectedundefined
HeadlessChrome 0.0.0 (Linux 0.0.0) ERROR
  Disconnectedundefined
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) DISCONNECTED (8 mins 17.695 secs / 6 mins 50.281 secs)
SUMMARY
  0: HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 2017 of 2123 (skipped 27) DISCONNECTED (8 mins 17.695 secs / 6 mins 50.281 secs)
test cases successful in all browsers

I’ve been looking into the suggestions posted in this issue: https://github.com/angular/angular/issues/12409

I’ve also looked into running the tests in Jest and also following the progress on Bazel very closely here: https://github.com/alexeagle/angular-bazel-example

[filipesilva]

We did some work to speed up rebuilds in 1.5.0 as @lucasvanhalst mentioned, but I know this is still a problem for many projects.

There’s a couple of parts to this issue:

• rebuild time: the time it takes for the CLI to rebuild bundles when a change is made.
• karma test run time: the time it takes for karma to finish tests after a refreshing.
• JIT compilation time: the time it takes for the JIT compiler to compile component templates at runtime.

Let’s look at them individually.

Rebuild time

This is the domain of the CLI build system. We have a highly customized karma webpack plugin that allows for a vendor chunk (https://github.com/angular/angular-cli/pull/6160).

This helps in rebuilds by allowing your test chunk to be rebuilt separately from the vendor chunk. Generally speaking, smaller chunks are faster to rebuild so it helps to split.

You can actually further split your test chunks as well using System.import() (or just import() if you use "module": "esnext" in your tsconfig https://github.com/Microsoft/TypeScript/issues/16675#issuecomment-309999772).

Modify your test files like this:

// end of ./src/test.ts

// // Then we find all the tests.
// const context = require.context('./', true, /\.spec\.ts$/);
// // And load the modules.
// context.keys().map(context);
// // Finally, start Karma to run the tests.
// __karma__.start();

// Import lazy test chunks.
declare var System: any;
Promise.all([
  System.import('./test-lazy-chunk-1.ts'),
  // More lazy chunk imports here.
]).then(() => {
  // Finally, start Karma to run the tests.
  __karma__.start();
});

// new file ./src/test-lazy-chunk-1.ts

// Then we find all the tests.
// Change the first argument of `.context()` to specify a subfolder like `./app/some-folder/`.
const context = (require as any).context('./', true, /\.spec\.ts$/);
// And load the modules.
context.keys().map(context);

// update ./src/tsconfig.spec.ts to include the lazy chunk

  "include": [
    "test-lazy-chunk-1.ts",
    "**/*.spec.ts",
    "**/*.d.ts"
  ]

Splitting your tests into chunk should make rebuilds faster.

Turning off sourcemaps also helps (by making rebundling faster) but then debugging will be hard.

Karma test run time

After a rebuild Karma will load your bundles and re-run all tests. So the more tests you have, the longer this will take. I think this is the problem most people actually have.

Jest solves this problem by only re-running stuff that needs to be run again. Last I checked we couldn’t really plug the webpack build setup (just try to convert) into it so it’s not an option.

The best you can do to limit this is limit the re-run tests via fdescribe of fit in your tests. Maybe we could run some advanced code transforms and do this for you? Would have to look into it.

JIT compilation time

After your code is rebuilt, it is loaded by the browser and tests are ran. Unit tests are always ran in JIT mode, which means that component templates are compiled in the browser.

This takes time. It’s fairly imperceptible in small projects but becomes noticeable as template complexity and number increase.

Outside of unit tests, this is the time after the rebuild is done and your browser has reloaded, but your screen is white and your application hasn’t actually loaded. That’s the JIT compiler processing templates.

The solution to reducing this time is to use AOT in unit tests, which means that the build step would also compile templates. This is still experimental so we haven’t added support to it in the CLI (see https://github.com/angular/angular-cli/pull/8007#issuecomment-336973766).

So for now there isn’t much that you can do to reduce this time (apart from limiting the specs as listed in the Karma test run time section above).