Parallel cypress test run failures now much more often
See original GitHub issueCurrent behavior
We have configured our cypress/react tests setup to run parallel in github actions across 6 different machines.
Just recently (after upgrading to 10.10.0) - we are now often seeing our test runs in cicd fail with the following response (> 50% of the time):
You passed the --parallel flag, but we do not parallelize tests across different environments.
This machine is sending different environment parameters than the first machine that started this parallel run.
The existing run is: https://dashboard.cypress.io/projects/[REDACTED]/runs
In order to run in parallel mode each machine must send identical environment parameters such as:
- specs
- osName
- osVersion
- browserName
- browserVersion (major)
This machine sent the following parameters:
{
"osName": "linux",
"osVersion": "Ubuntu - 20.04",
"browserName": "Chrome",
"browserVersion": "107.0.[53](https://github.com/immutable/imx-marketplace-next/actions/runs/3382557523/jobs/5617596861#step:5:54)04.87",
"specs": [
REDACTED
]
}
https://on.cypress.io/parallel-group-params-mismatch
Test run failed, code 1
More information might be available above
Cypress module has returned the following error message:
Could not find Cypress test run results
Error: Could not find Cypress test run results
Until 10.10 - all of these tests were passing 100% in parallel in cicd. We are also seeing the same thing happen in 10.11.0
Meanwhile, we if view this above cicd test run in cypress dashboard, all tests are shown as passing! 🤪
Desired behavior
All tests should run and pass if they are valid, in parallel in cicd
Test code to reproduce
n/a
Cypress Version
10.11.0
Node version
16.18.0
Operating System
“ubuntu-latest”
Debug Logs
No response
Other
No response
Issue Analytics
- State:
- Created a year ago
- Reactions:1
- Comments:7
Top Results From Across the Web
Parallelization | Cypress Documentation
Cypress can run recorded tests in parallel across multiple machines since version 3.1.0. While parallel tests can also technically run on a single...
Read more >Cypress testing: Running tests in parallel - Valor Software
Its main task is to check how the client part interacts with the server ... Cypress tries to check it a few times,...
Read more >Cypress Parallelization — Case Study + Results - Medium
All programmers are faced with bugs. If your team ignores tests or you personally do not test the code, issues in the project...
Read more >How to Run Cypress Tests in Parallel | BrowserStack
Before diving into how to run parallel tests in Cypress, let's get a quick overview of parallel testing in general · Parallel Testing...
Read more >Is there any way to run parallel cypress tests on local machine ...
Technically, it's possible. Cypress does not recommend it since running multiple cypress instances on the same machine consumes lots of ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I’ve run into this issue before.
Your failures are caused by this change: https://github.com/actions/runner-images/commit/0c8b93e28a4076a77df756bbb3f69ac72bceca9f
GitHub periodically updates Chrome to the latest version on their action runners. The runners aren’t all updated at the same time. They are upgraded over a period of a few days. You can’t control which action runner versions your workflow uses.
If you’re running Cypress tests in parallel on multiple runners during such an upgrade phase, you can end up with different runners using different Chrome versions. This is the error you get when that happens.
However, when Cypress fails like this, the tests simply do not get run on that runner. What happens instead is that they remain in the queue and end up being picked up later by another runner. That’s why your dashboard is still green.
@cgraham-rs You may definitely be installing the latest version of Chromium (109 there), but are you sure Cypress is actually using it? Because your Cypress logs still show your tests using Chrome 106 and 107. Don’t you need to specify your browser to be
chromium
or the full path to the browser you installed?One way of working around this problem is to not let your CI depend on the results of your individual parallel runners. Aggregate the results of the parallel runners in a separate step and pass/fail based on that. Individual runners can still fail, but Cypress will eventually run all your tests.
At worst, you end up in a situation where the first runner that started is the only runner on the old/new Chrome and it ends up having to run all your tests because none of the other runner environments match.