Slow startup time in monorepos (single test/project)

All of this is done on OSX with 3,5 GHz Dual-Core Intel Core i7 and 1 jest worker.

Problem

We have a monorepo with currently around 26 packages. We currently run with yarn (1) workspaces. So the code base is not exactly small, but once you work on it you mostly work on one of those packages at a time. The problem is that even when just running a single test, it takes about ~10 seconds for the tests to finish. This is mostly startup time because jest reports the test itself running in ~100ms.

We would like to get this time down to allow for a better developer experience. The fact that all tests together take almost ten minutes doesn’t bother us that much, but running a single test should ideally finish in less than a second.

We hope that someone here can help us or at least point us in the right direction.

jest config

module.exports = {
  testRunner: 'jest-circus/runner',
  transform: {
    '^.+\\.(t|j)sx?$': 'babel-jest',
  },
  transformIgnorePatterns: ['<rootDir>/node_modules/'],
  moduleNameMapper: {
    '//': 'Here are 10 modules mapped',
  },
  clearMocks: true,
  roots: ['<rootDir>'],
  snapshotSerializers: ['jest-date-serializer'],
  testURL: 'http://www.test.com',
  moduleFileExtensions: ['tsx', 'ts', 'js', 'json'],
  testEnvironment: 'jest-environment-jsdom-sixteen',
  setupFilesAfterEnv: [
    'jest-canvas-mock',
    'jest-localstorage-mock',
    '<rootDir>/setup/index.ts',
    '<rootDir>/setup/errorCatcher.ts',
  ],
  reporters: ['default', '<rootDir>/setup/consoleErrorReporter.js'],
  rootDir: '<rootDir>/../../../',
};

consoleErrorReporter gathers information about errors written to the console.

So in order to allow a custom config for each project the final config is dynamically built:

const fs = require('fs');
const { rootDir, ...baseConfig } = require('./setup/baseJestConfig');
const packages = []; // we have a function which returns all packages

const projects = packages.map(({ location, title }) => {
  try {
    // use config if provided by a package
    fs.accessSync(`${location}/jest.config.js`);
    return `<rootDir>/${location}`;
  } catch (e) {
    // otherwise just use the base config
    return {
      ...baseConfig,
      displayName: title,
      testRegex: `${location}/.*(Test|.test)\\.(t|j)sx?$`,
    };
  }
});

module.exports = {
  ...baseConfig,
  roots: ['<rootDir>'],
  projects,
};

In the end projects would be an array of 26 configs, which mostly look the same.

What we tried so far

Different transpiler

I first thought transpilation might be a bottleneck. I tried swc and esbuild. To my surprise, it made no difference.

Define just the config for the package you are using

We initially filtered for configs we need for a run, but then found out about --selectedProjects. Both approaches sped up startup time by a factor of three on my machine. My colleague (with slightly better hardware) could observe around 50% speedup, regardless of the total amount of tests that he ran.

How we tried to debug

Get some times

Hacked timings in jest-runtime/build/index.js like

console.time(module.filename);
compiledFunction.call(
  module.exports,
  module, // module object
  module.exports, // module exports
  module.require, // require implementation
  module.path, // __dirname
  module.filename, // __filename
  this._environment.global, // global object
  ...lastArgs.filter(notEmpty)
);
console.timeEnd(module.filename);

Most files take less than a ms, longest took around 600ms. I can see this pilling up for 7-8k files when done in sync.

Profiled the node process

We are not very familiar with how to read and interpret these reports. Here is an excerpt from it, I cut off lines and just left the top 5 for each:

Statistical profiling result from isolate-0x10469d000-91221-v8.log, (20504 ticks, 27 unaccounted, 0 excluded).

 [Shared libraries]:
   ticks  total  nonlib   name
    300    1.5%          /usr/lib/system/libsystem_platform.dylib
     65    0.3%          /usr/lib/system/libsystem_pthread.dylib
     47    0.2%          /usr/lib/system/libsystem_kernel.dylib
     25    0.1%          /usr/lib/system/libsystem_malloc.dylib
      1    0.0%          /usr/lib/system/libdispatch.dylib

 [JavaScript]:
   ticks  total  nonlib   name
    116    0.6%    0.6%  RegExp: /\.git/|/\.hg/
     53    0.3%    0.3%  LazyCompile: *_ignore /Users/****/node_modules/jest-haste-map/build/index.js:1191:10
     30    0.1%    0.1%  LazyCompile: *<anonymous> /****/node_modules/jest-haste-map/build/crawlers/node.js:254:15
     22    0.1%    0.1%  RegExp: .*\/locales\/.*en\.json$
     14    0.1%    0.1%  LazyCompile: *resolve path.js:973:10

 [C++]:
   ticks  total  nonlib   name
   8465   41.3%   42.2%  T __kernelrpc_thread_policy_set
   3148   15.4%   15.7%  T __ZN2v88internal19ScriptStreamingDataC2ENSt3__110unique_ptrINS_14ScriptCompiler20ExternalSourceStreamENS2_14default_deleteIS5_EEEENS4_14StreamedSource8EncodingE
   2265   11.0%   11.3%  T node::SyncProcessRunner::Spawn(v8::FunctionCallbackInfo<v8::Value> const&)
   1135    5.5%    5.7%  T __kernelrpc_mach_vm_purgable_control_trap
    596    2.9%    3.0%  t node::fs::Read(v8::FunctionCallbackInfo<v8::Value> const&)

 [Summary]:
   ticks  total  nonlib   name
    439    2.1%    2.2%  JavaScript
  19600   95.6%   97.7%  C++
    487    2.4%    2.4%  GC
    438    2.1%          Shared libraries
     27    0.1%          Unaccounted

 [C++ entry points]:
   ticks    cpp   total   name
   4961   47.8%   24.2%  T __ZN2v88internal21Builtin_HandleApiCallEiPmPNS0_7IsolateE
   3711   35.8%   18.1%  T __ZN2v88internal19ScriptStreamingDataC2ENSt3__110unique_ptrINS_14ScriptCompiler20ExternalSourceStreamENS2_14default_deleteIS5_EEEENS4_14StreamedSource8EncodingE
    933    9.0%    4.6%  T __kernelrpc_mach_vm_purgable_control_trap
    130    1.3%    0.6%  T __ZN2v88internal30Builtin_ErrorCaptureStackTraceEiPmPNS0_7IsolateE
    129    1.2%    0.6%  T _open$NOCANCEL

Interestingly the CPU profiler in node shows a lot (~4s) of “nothing” in between starting the script and executing jest:

Also the jestAdapter takes 8s before a tests starts and a total of 12s for the entire run.

Test Suites: 5 passed, 5 total
Tests:       26 passed, 26 total

As far as I can tell the “nothing” time is spent with reading files. onStreamRead and program, zoomed in:

Issues that might be related

https://github.com/facebook/jest/issues/10301 https://github.com/facebook/jest/issues/9554

[joshribakoff-sm]

@mingshenggan as described in my above comment(s) it’s because jest recursively follows all import statements during startup. So by importing 1 MUI icon via the barrel export, it forces jest to crawl 1000s of file(s) also imported by that barrel export.

Certainly reducing the amount of imports will help, but assuming your project actually needs to import part(s) of larger libraries, it doesn’t solve the root issue within jest

[gor918]

const jest: { globals: { "ts-jest": { "isolatedModules": true } }, }

Testing was sped up with this configuration