Parallel cypress test run failures now much more often

Current behavior

We have configured our cypress/react tests setup to run parallel in github actions across 6 different machines.

Just recently (after upgrading to 10.10.0) - we are now often seeing our test runs in cicd fail with the following response (> 50% of the time):

You passed the --parallel flag, but we do not parallelize tests across different environments.

This machine is sending different environment parameters than the first machine that started this parallel run.

The existing run is: https://dashboard.cypress.io/projects/[REDACTED]/runs

In order to run in parallel mode each machine must send identical environment parameters such as:

 - specs
 - osName
 - osVersion
 - browserName
 - browserVersion (major)

This machine sent the following parameters:

{
  "osName": "linux",
  "osVersion": "Ubuntu - 20.04",
  "browserName": "Chrome",
  "browserVersion": "107.0.[53](https://github.com/immutable/imx-marketplace-next/actions/runs/3382557523/jobs/5617596861#step:5:54)04.87",
  "specs": [
      REDACTED
  ]
}

https://on.cypress.io/parallel-group-params-mismatch
Test run failed, code 1
More information might be available above
Cypress module has returned the following error message:
Could not find Cypress test run results
Error: Could not find Cypress test run results

Until 10.10 - all of these tests were passing 100% in parallel in cicd. We are also seeing the same thing happen in 10.11.0

Meanwhile, we if view this above cicd test run in cypress dashboard, all tests are shown as passing! 🤪

Desired behavior

All tests should run and pass if they are valid, in parallel in cicd

Test code to reproduce

n/a

Cypress Version

10.11.0

Node version

16.18.0

Operating System

“ubuntu-latest”

Debug Logs

No response

Other

No response

[gofr]

I’ve run into this issue before.

Your failures are caused by this change: https://github.com/actions/runner-images/commit/0c8b93e28a4076a77df756bbb3f69ac72bceca9f

GitHub periodically updates Chrome to the latest version on their action runners. The runners aren’t all updated at the same time. They are upgraded over a period of a few days. You can’t control which action runner versions your workflow uses.

If you’re running Cypress tests in parallel on multiple runners during such an upgrade phase, you can end up with different runners using different Chrome versions. This is the error you get when that happens.

However, when Cypress fails like this, the tests simply do not get run on that runner. What happens instead is that they remain in the queue and end up being picked up later by another runner. That’s why your dashboard is still green.

[gofr]

@cgraham-rs You may definitely be installing the latest version of Chromium (109 there), but are you sure Cypress is actually using it? Because your Cypress logs still show your tests using Chrome 106 and 107. Don’t you need to specify your browser to be chromium or the full path to the browser you installed?

One way of working around this problem is to not let your CI depend on the results of your individual parallel runners. Aggregate the results of the parallel runners in a separate step and pass/fail based on that. Individual runners can still fail, but Cypress will eventually run all your tests.

At worst, you end up in a situation where the first runner that started is the only runner on the old/new Chrome and it ends up having to run all your tests because none of the other runner environments match.