CI: Sporadic failures in example-firefox

Describe the bug

.github/workflows/example-firefox.yml often fails.

Typically the failure logs show:

Still waiting to connect to Firefox, retrying in 1 second
Warning: We failed processing this video.
Cypress failed to make a connection to Firefox.
Could not find Cypress test run results
Error: Could not find Cypress test run results

Possibly also

Error: connect ECONNREFUSED 127.0.0.1:38171
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)

Bug history

View workflow logs via https://github.com/cypress-io/github-action/actions/workflows/example-firefox.yml?query=branch%3Amaster

Job Id	Failure job	Firefox version
firefox	6185401336	107
firefox-v10	6056300428	105

The workflow has been failing on merges into the master branch since November 22, 2022. Problems with merges into other branches have been going on for much longer. Due to log retention policies only recent detailed logs going back 90 days are available. Job 5079650276 from September 26, 2022 is the earliest concrete evidence of the exact same failure with Firefox 104.

Steps to reproduce the issue

Open a pull request to trigger a run of .github/workflows/example-firefox.yml

View the results on https://github.com/cypress-io/github-action/actions/workflows/example-firefox.yml

Expected behavior

All sub-tests of .github/workflows/example-firefox.yml should run reliably and succeed.

Comments

I could not find a reason for the unreliable results of .github/workflows/example-firefox.yml so far.

It would be possible to stop the following errors being reported:

• “Warning: We failed processing this video.” (disable videos)
• “Warning: No files were found with the provided path: examples/firefox/cypress/screenshots. No artifacts will be uploaded” in the firefox job for v9 (correct the directory)

however this does not make the action more reliable.

[MikeMcC399]

• PR #660 has resolved this issue through the workaround of skipping the problematic Firefox test with Cypress v9. Issue https://github.com/cypress-io/cypress/issues/23215 provides background information about why no other resolution is being pursued.

• The action log 3705934530 shows the successful result of skipping the Firefox v9 test, which allows the Firefox v10 test to be successful.

Thanks to @jaffrepaul and @nagash77 for supporting the resolution of this issue!

[MikeMcC399]

I got today’s new release of Firefox 108 and tried on Windows 11, with Cypress 9.7.0 (without GHA).

If I start a separate instance of Firefox then execute npx cypress run --browser firefox --headed I get no problem, however if I then close all Firefox sessions and execute the above command again, I get Error: could not find CRI target

This is an interoperability problem between Cypress 9.7.0 and Firefox 108. I can’t say which side of the two connection partners is at fault, but it certainly isn’t github-action, since I’m able to reproduce the issue without it.

The test script is:

/// <reference types="Cypress" />

it('works', () => {
  cy.visit('https://example.cypress.io')
})