Add FAQ for why Cypress passes locally and not in CI.

I’m submitting a…

[ ] Bug report
[x] Content update
[ ] Process update (build, deployment, ... )

Type of bug / changes

It is pretty common question / complain from users and deserves an answer.

Some saved responses in the past:

1

There are a lot of reasons tests could fail in CI and pass locally. On top of that - running in CI is a bit of a black box. So, the combination of these 2 facts make it especially frustrating to debug failing tests that solely fail in CI.

Some reasons this could happen:

1. There is a problem isolated to the Chrome version of our Electron browser. We’re working on updating it every day. https://github.com/cypress-io/cypress/pull/4720 Installing a later Chrome version in Travis would highlight if this is your problem.
2. If there is a build process in between running tests locally and running in CI - sometimes Cypress test failures are highlighting a bug in the build process.
3. Variability in timings when running your application in CI. If you have a test that will only pass when a network request finishes in 10ms (which happens every time on your local machine) and then that request takes 20ms on your CI machine (for whatever reason from that machine - location, CPU, etc slows it down) - this can cause some unexpected failures. You’ll want to make sure to write assertions around the important steps that need to be reached before moving on to the next action in your tests. for example - ensuring a network request has finished before looking for the DOM element that relies on the data from that network request. I would suggest reviewing the video of the test failures, looking at the timings of commands and actions in the Command Log, and comparing them to your local run’s Command Log to highlight if this is the issue.

2

Tests run in CI run within the Electron browser. So, the first suggestion is to run the tests locally within the Electron browser with DevTools open to see if there are any issues there.

Since you already mentioned running the tests locally with other browser combinations, next I would make sure that video recording and/or screenshots are turned on during the CI run. Review the video to see if anything can be detected there that is causing the failure.

If it’s still not obvious, compare the Command Log from within the video to the Command Log of the same test passing locally when run in Cypress. Take a screenshot of each at the same moment and compare each command side by side. Seeing differences of the execution time or missing commands, etc can sometimes illuminate differences in the timings of requests or other methods that may have caused an issue.

3 (more about flaky tests)

Most often in cases of flaky tests, we see that there is not enough assertions surrounding the actions or XHR requests/responses necessary before moving on to the next assertion. So if there is any variation in the speed of the XHR requests response in interactive mode versus run mode, then there will start to be failures in one over the other - or flaky tests.

So, for example if you had a test to check that when you click a ‘Save’ button that the users page refreshes with the correct content, you may write a test like so:

cy.get(‘.name-input’).type(‘Jennifer’) cy.get(‘form’).contains(‘Save’).click() cy.get('#user- name).should(‘contain’, ‘Jennifer’)

Each command by default will wait 4 seconds to meet it’s requirements, but if the POST for the form request sometimes takeds longer to respond than it takes for the “cy.get('#user- name).should(‘contain’, ‘Jennifer’)” to run, then it will fail sometimes. Because of this, we recommend asserting on as many required steps as possible - this also helps later to isolate where the exact failure is when debugging on an actual bug.

So, the example above would be less flaky written like this:

cy.server() cy.route(‘POST’, ‘users/*’).as(‘updateUser’) cy.route(‘GET’, ‘users’).as(‘getUsers’)

cy.get(‘.name-input’).type(‘Jennifer’) cy.get(‘form’).contains(‘Save’).click() cy.wait(‘@updateUser’).then((xhr) => { expect(xhr.requestBody).to.have.property(‘name’, ‘Jennifer’) } cy.url(‘contains’, ‘users/123’) cy.wait(‘@getUsers’) cy.get('#user- name).should(‘contain’, ‘Jennifer’)

Now when the last line runs, we don’t have to worry about the POST or that the POST sent the right information, that the url changed and the GET’s or whatever requests needed to respond - we know it’s in the right state to find the content here.

[RabinMallick]

Facing same issue in 2022. Tests are passing locally but fails in Bitbucket pipeline. Any promising solution?

[daryamatsenko]

Experiencing the same issue: tests always passed locally but failing in circleci. Did anybody find a solution for this?