Parallelizing across machines in CI

Many CI systems support splitting up a single task across multiple machines. This can drastically speed up CI times even when you are duplicating some of the work compiling or whatever.

From what I’ve seen, they mostly work like this:

steps:
  - command: yarn install && yarn test
    parallelism: 3

# on machine 1
CI_NODE_TOTAL=3 CI_NODE_INDEX=0 yarn install && yarn test

# on machine 2
CI_NODE_TOTAL=3 CI_NODE_INDEX=1 yarn install && yarn test

# on machine 3
CI_NODE_TOTAL=3 CI_NODE_INDEX=2 yarn install && yarn test

For example: https://buildkite.com/docs/builds/parallel-builds

It would be awesome if there was an easy way to integrate this with Jest such that you could automatically chunk tests up so that they can be run across machines.

I imagine you’d have to discover all the tests and then have some stable way of splitting them up, either by counting the tests or by splitting the files up.

I think you could even automatically do this without any additional setup from developers. I created a module to help you do just that: https://github.com/jamiebuilds/ci-parallel-vars

---

We do this in our project today by starting 4 separate Jest runs on different machines:

steps:
  - CI_NODE_TOTAL=4 CI_NODE_INDEX=0 JEST_TESTS=$(jest --listTests --json) jest
  - CI_NODE_TOTAL=4 CI_NODE_INDEX=1 JEST_TESTS=$(jest --listTests --json) jest
  - CI_NODE_TOTAL=4 CI_NODE_INDEX=2 JEST_TESTS=$(jest --listTests --json) jest
  - CI_NODE_TOTAL=4 CI_NODE_INDEX=3 JEST_TESTS=$(jest --listTests --json) jest

Then in jest.config.js we split up the tests:

let parallelism = require('ci-parallel-vars')
let chunkd = require('chunkd')

let tests = JSON.parse(process.env.JEST_TESTS).sort((a, b) => {
  return b.localeCompare(a)
})

if (parallelism) {
  tests = chunkd(tests, parallelism.index, parallelism.total)
}

module.exports = {
  testMatch: tests,
}

This sped up our builds significantly

[aaronabramov]

i was talking about something else (host process and a bunch of remote clients), but i think it would be nice to do jest --chunk=2/10 kind of stuff in Jest

[thiamsantos]

I would love to see this feature been native to jest. Right now we have been using the approach suggested at https://github.com/facebook/jest/issues/11252#issuecomment-813494558. But seems to be a common enough functionality to be added to jest itself. When a project reaches a certain size is nice to be able to split across different jobs/machines in CI.

Seems that we have a few suggestions on how the API would look like:

• Parallel vars: CI_NODE_TOTAL=3 CI_NODE_INDEX=0 yarn test
• Chunk: yarn test --chunk=0/3
• Shard from #11252 : jest --shard-from=0 --shard-to=0.2

I would like to add another suggestion, based on elixir test framework:

JEST_PARTITION=2 yarn test --partitions 3 or yarn test --total-partitions 4 --partition 1

---

I would be happy to send a PR if the feature is welcome.