Accept flag `--parallel` to run spec files in parallel

What users want

To run spec files in parallel in order to speed up test runs (particularly when in CI)

How we will do this

Specs will have the option of running in parallel by specifying a --parallel flag in cypress run.

Spec files will run in parallel when --parallel flag passed and when determined to be within the same homogeneous environment (determined by Cypress by determining the runs ci-group and group).

• ci-group - is defined by the CI provider (a unique id per build usually, like CI_BUILD_NUM, it can also be specified by the flag ci-group-id if Cypress is unable to determine one). All this means is that a “run” in Cypress is the equivalent of a “run/build/workflow” in your CI provider. TLDR: Basically, you can’t parallelize tests across different CI providers or different CI runs.
• group - within a single ci-group, it is determined by taking into account the uniqueness of the combination of OS name, OS version, browser name, browser version, and defined specs. Groups can be defined yourself and named, but this is optional. Groups will be figured out automatically by Cypress otherwise.

We will automatically balance each group’s specs by machine based on forecasting we have calculated based off of their previous run data:

• Machine A gets Spec 5,9,10
• Machine B gets Spec 4,1
• Machine C gets Spec 6,11,8,2
• Machine D gets Spec 3,7

Each machine will run at the same time and simply ask “for the next spec file”. That means the work completes as fast as possible without needing to manually balance or tweak anything.

This will allow you to test files and have them parallelized within one run like:

my CI .yml file

// job 1
cypress run --record --parallel

Cypress will parallelize all spec files within the project across all available machines.

This will allow me to test several apps (like a monorepo) within one run and group them like:

my CI .yml file

// job 1
cypress run --group=app-1 --config integrationFolder=app-1/cypress/integration
// job 2
cypress run --group=app-2 --config integrationFolder=app-2/cypress/integration

This will allow me to test several environments within one run and group them like:

my CI .yml file

// job 1
cypress run --group=development --config=baseUrl=http://develop-app.com --env DEVELOP=true
// job 2
cypress run --group=staging --config=baseUrl=http://staging-app.com --env DEVELOP=true

What you will see

• The Dashboard Service will display the specs within a run grouped by `group
• Each failure will show which group it is in
• You will be able to visualize exactly how the specs ran across the machines and also visualize how any future runs will be affected by adding or removing machines.

Dashboard Service comps

[Toxicable]

Would it be possible to add a simple and easy to use degree of parallelism flag? ie: I want to run 3 specs at the same time: --parallelism=3 or I want to run all specs at the same time: --parallelism=all, where all = number of cores - 2

This will make it much easier for Cypress to utilise all resources on whatever system it’s running on. Without adding the extra complexity of having to refactor all your specs into different directories and adding the groups proposed.

But I still think it’s important to have both options avilable to fit all use cases

[brian-mann]

@Toxicable Cores here isn’t really the problem, memory is likely the bottleneck. How much memory do your machines have?

Also are they linux machines? Browsers behave differently when they are out a focus, and only a single browser can be in focus… but in linux with the use of xvfb they will behave as if they have focus (which is what we do today).

Regardless though - I believe our implementation will work flawlessly for you. It’s up to you to determine how many machines you want to run in parallel - and you can easily do this with a node script or even a bash command.

const _ = require('lodash')
const os = require('os')
const cypress = require('cypress')

// kick off a bunch of cypress runs
const runs = _.times(os.cpus.length / 2, (i) => {
  return cypress.run({ record: true, parallel: true }) // this is it
})

Promise.all(runs)
.then((resultsArray) => {
  const totalFailed = _.map(resultsArray, "totalFailed")

  process.exit(totalFailed)
})

This kicks off a bunch of cypress runs based on half of your cpu cores. You may have other problems trying to run this all on a single machine. For instance your server or database would have to support multiple sesssions at the same time. If you do any kind of seeding you will have to take into account.

That’s why we generally suggest parallelizing at the OS level. For instance, you could take your beefy machine and cut it into a dozen docker containers running all at the same time. It would work the same way. In most CI providers, that’s how it works - they are all isolated from each other.

The --parallel flag automatically load balances the specs, you don’t have to specify anything.

The grouping as @jennifer-shehane suggested is only if you want to chunk your specs by a particular group and then manage / parallelize them differently. If you have a single group, then you don’t do anything.

EDIT: I just realized that spawning a bunch of cypress runs together will clobber things like screenshotsFolder and videosFolder. You could pass a configuration option using the (i) index to create those dynamically.

Also, I think there may be an issue because we don’t create dynamic user data dir profiles for chrome or electron. There’s already an issue open with that, and that would enable chrome to be opened multiple times and prevent session clashing with cookies or localstorage. That’s a really simple, easy fix.