Running with Cypress is flaky in CircleCi

Goto https://github.com/bahmutov/start-server-and-test/issues/250#issuecomment-1079451745 to see how I resolved this

Running Cypress will intermittently fail in CircleCI. I haven’t been able to identify a pattern that causes it, my guess is it’s some kind of race condition.

Running yarn cy:ci locally works perfectly every time.

in package.json:

"scripts": {
    ...
    "start": "ng serve",
    "cy:run": "$(yarn bin)/cypress run",
    "cy:ci": "start-server-and-test start http-get://localhost:4200 cy:run",
    ...
  },
  ...
  "devDependencies": {
   ...
    "cypress": "^4.0.0",
    "cypress-browser-extension-plugin": "^0.1.0",
    "cypress-iframe": "^1.0.1",
    "start-server-and-test": "^1.10.11",
   ...
    },

in .circleci/config.yml:

      - run:
          name: 'Run Cypress tests'
          command: yarn cy:ci

circle logs:

#!/bin/bash -eo pipefail
yarn cy:ci
yarn run v1.22.4
$ start-server-and-test start http-get://localhost:4200 cy:run
starting server using command "npm run start"
and when url "[ 'http-get://localhost:4200' ]" is responding with HTTP status code 200
running tests using command "npm run cy:run"

> app@0.0.0 start /home/circleci/repo
> ng serve

... // App builds


** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **
: Compiled successfully.

> app@0.0.0 cy:run /home/circleci/repo
> $(yarn bin)/cypress run

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! app@0.0.0 cy:run: `$(yarn bin)/cypress run`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the app@0.0.0 cy:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2020-04-15T21_27_08_984Z-debug.log
Error: Command failed with exit code 1: npm run cy:run
    at makeError (/home/circleci/repo/node_modules/start-server-and-test/node_modules/execa/lib/error.js:56:11)
    at handlePromise (/home/circleci/repo/node_modules/start-server-and-test/node_modules/execa/index.js:114:26)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  command: 'npm run cy:run',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: undefined,
  stderr: undefined,
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Exited with code exit status 1
CircleCI received exit code 1

complete log:

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'run', 'cy:run' ]
2 info using npm@6.14.4
3 info using node@v13.12.0
4 verbose run-script [ 'precy:run', 'cy:run', 'postcy:run' ]
5 info lifecycle app@0.0.0~precy:run: app@0.0.0
6 info lifecycle app@0.0.0~cy:run: app@0.0.0
7 verbose lifecycle app@0.0.0~cy:run: unsafe-perm in lifecycle true
8 verbose lifecycle app@0.0.0~cy:run: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/circleci/repo/node_modules/.bin:/usr/local/bin:/tmp/yarn--1587066858616-0.6618809648036963:/home/circleci/repo/node_modules/.bin:/home/circleci/.config/yarn/link/node_modules/.bin:/home/circleci/.yarn/bin:/usr/local/libexec/lib/node_modules/npm/bin/node-gyp-bin:/usr/local/lib/node_modules/npm/bin/node-gyp-bin:/usr/local/bin/node_modules/npm/bin/node-gyp-bin:/home/circleci/.local/bin:/home/circleci/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
9 verbose lifecycle app@0.0.0~cy:run: CWD: /home/circleci/repo
10 silly lifecycle app@0.0.0~cy:run: Args: [ '-c', '$(yarn bin)/cypress run' ]
11 silly lifecycle app@0.0.0~cy:run: Returned: code: 1  signal: null
12 info lifecycle app@0.0.0~cy:run: Failed to exec cy:run script
13 verbose stack Error: app@0.0.0 cy:run: `$(yarn bin)/cypress run`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack     at EventEmitter.emit (events.js:315:20)
13 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:315:20)
13 verbose stack     at maybeClose (internal/child_process.js:1026:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
14 verbose pkgid app@0.0.0
15 verbose cwd /home/circleci/repo
16 verbose Linux 4.15.0-1052-aws
17 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "cy:run"
18 verbose node v13.12.0
19 verbose npm  v6.14.4
20 error code ELIFECYCLE
21 error errno 1
22 error app@0.0.0 cy:run: `$(yarn bin)/cypress run`
22 error Exit status 1
23 error Failed at the app@0.0.0 cy:run script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

It seems that everything works fine until the cy:run script runs. I have no idea why it fails randomly. Any ideas?

[diogob]

@dominikwilkowski I know your comment was long ago, but just for future developers reading this thread, I had a very similar issue and, in my case, upcon closer inspection I found and error ENOSPC: System limit for number of file watchers reached.

Then I successfully fixed the issue following the advice from this stack overflow answer

[AleksanderBodurri]

Sorry to revive after so long. I’m pretty sure I have a good idea of why this issue was occurring, at least in my case. I was writing tests that did not take full advantage of cypress’ retry-ability mechanism. I was writing tests that looked like this:

cy.get('.foo').find('.bar').contains('baz')

Heres the scenario:

1. A view exists like this

<div class="foo">
  <div class="bar">
        qux
  </div>
</div>

1. A cypress command causes some event that triggers the view to update. After the update, the view should look like

<div class="foo">
  <div class="bar">
        baz
  </div>
</div>

Note that the divs after the update are not the same DOM elements as the divs before the update.

1. Cypress then runs cy.get('.foo').find('.bar').contains('baz'). If the page view doesn’t change quickly enough (as is sometimes the case in CI containers where system resources are not as high as our local devices), the cy.get('.foo').find('.bar') part of the query matches against the elements from step 1. Since these elements will never contain 'baz', even if the contains('baz') is retried after the view updates, cy.get('.foo').find('.bar').contains('baz') will fail.

For some reason the details of the test failures were being silenced. Not sure exactly why that was happening, but I haven’t seen the same flakiness since I refactored my tests to avoid the scenario above.

The biggest thing that helped me fix this issue was a snippet from the Cypress docs here https://docs.cypress.io/guides/core-concepts/retry-ability#Merging-queries

Specifically

Use cy.contains Tip: instead of cy.get(selector).should(‘contain’, text) or cy.get(selector).contains(text) chain, we recommend using cy.contains(selector, text) which is retried automatically as a single command.

Documentation is once again undefeated 😄