question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Parallel cypress test run failures now much more often

See original GitHub issue

Current behavior

We have configured our cypress/react tests setup to run parallel in github actions across 6 different machines.

Just recently (after upgrading to 10.10.0) - we are now often seeing our test runs in cicd fail with the following response (> 50% of the time):

You passed the --parallel flag, but we do not parallelize tests across different environments.

This machine is sending different environment parameters than the first machine that started this parallel run.

The existing run is: https://dashboard.cypress.io/projects/[REDACTED]/runs

In order to run in parallel mode each machine must send identical environment parameters such as:

 - specs
 - osName
 - osVersion
 - browserName
 - browserVersion (major)

This machine sent the following parameters:

{
  "osName": "linux",
  "osVersion": "Ubuntu - 20.04",
  "browserName": "Chrome",
  "browserVersion": "107.0.[53](https://github.com/immutable/imx-marketplace-next/actions/runs/3382557523/jobs/5617596861#step:5:54)04.87",
  "specs": [
      REDACTED
  ]
}

https://on.cypress.io/parallel-group-params-mismatch
Test run failed, code 1
More information might be available above
Cypress module has returned the following error message:
Could not find Cypress test run results
Error: Could not find Cypress test run results

Until 10.10 - all of these tests were passing 100% in parallel in cicd. We are also seeing the same thing happen in 10.11.0

Meanwhile, we if view this above cicd test run in cypress dashboard, all tests are shown as passing! 🤪

Desired behavior

All tests should run and pass if they are valid, in parallel in cicd

Test code to reproduce

n/a

Cypress Version

10.11.0

Node version

16.18.0

Operating System

“ubuntu-latest”

Debug Logs

No response

Other

No response

Issue Analytics

  • State:closed
  • Created a year ago
  • Reactions:1
  • Comments:7

github_iconTop GitHub Comments

2reactions
gofrcommented, Nov 3, 2022

I’ve run into this issue before.

Your failures are caused by this change: https://github.com/actions/runner-images/commit/0c8b93e28a4076a77df756bbb3f69ac72bceca9f

GitHub periodically updates Chrome to the latest version on their action runners. The runners aren’t all updated at the same time. They are upgraded over a period of a few days. You can’t control which action runner versions your workflow uses.

If you’re running Cypress tests in parallel on multiple runners during such an upgrade phase, you can end up with different runners using different Chrome versions. This is the error you get when that happens.

However, when Cypress fails like this, the tests simply do not get run on that runner. What happens instead is that they remain in the queue and end up being picked up later by another runner. That’s why your dashboard is still green.

1reaction
gofrcommented, Nov 4, 2022

@cgraham-rs You may definitely be installing the latest version of Chromium (109 there), but are you sure Cypress is actually using it? Because your Cypress logs still show your tests using Chrome 106 and 107. Don’t you need to specify your browser to be chromium or the full path to the browser you installed?

One way of working around this problem is to not let your CI depend on the results of your individual parallel runners. Aggregate the results of the parallel runners in a separate step and pass/fail based on that. Individual runners can still fail, but Cypress will eventually run all your tests.

At worst, you end up in a situation where the first runner that started is the only runner on the old/new Chrome and it ends up having to run all your tests because none of the other runner environments match.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Parallelization | Cypress Documentation
Cypress can run recorded tests in parallel across multiple machines since version 3.1.0. While parallel tests can also technically run on a single...
Read more >
Cypress testing: Running tests in parallel - Valor Software
Its main task is to check how the client part interacts with the server ... Cypress tries to check it a few times,...
Read more >
Cypress Parallelization — Case Study + Results - Medium
All programmers are faced with bugs. If your team ignores tests or you personally do not test the code, issues in the project...
Read more >
How to Run Cypress Tests in Parallel | BrowserStack
Before diving into how to run parallel tests in Cypress, let's get a quick overview of parallel testing in general · Parallel Testing...
Read more >
Is there any way to run parallel cypress tests on local machine ...
Technically, it's possible. Cypress does not recommend it since running multiple cypress instances on the same machine consumes lots of ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found