Cypress fails to retry job on GitLab CI with --record and --group flags

Current behavior:

On GitLab CI, when a recorded job with multiple groups fails and is being retried, Cypress errors out with You passed the --group flag, but this group name has already been used for this run. for the second group.

Desired behavior:

Retries should run for all groups.

Test code to reproduce

In our particular case we run two groups:

// .gitlab-ci.yml

cypress:
  stage: test
  image: cypress/browsers:node14.7.0-chrome84
  before_script:
    - cd cypress
    - npm ci
  script:
    - npm run cli -- --browser chrome --headless --record --group Backend --spec 'integration/backend/**/*'
    - npm run cli -- --browser chrome --headless --record --group Frontend --spec 'integration/frontend/**/*'
  artifacts:
    when: always
    paths:
      - cypress/videos/**/*.mp4
      - cypress/screenshots/**/*.png
    expire_in: 1 day

The first run fails for whatever reason. The second run (retry) starts, executes the first group (Backend) without issue, and fails on the second group (Frontend).

Second run, first group:

Second run, second group: Here you can see Recorded Run: https://dashboard.cypress.io/projects/khj9rq/runs/16 for the first group, followed by You passed the --group flag, but this group name has already been used for this run. for the second group:

Adding a new commit to the branch “fixes” it and lets the tests pass again.

On the Dashboard it translates to incomplete or mismatched reporting:

Incomplete run 16 from above logs:

Incomplete previous run 15:

Successful next run 17:

Versions

• “cypress”: “5.0.0”
• gitlab-runner 13.0.0

[damusnet]

Hi @bahmutov, thanks for the quick investigation! I figured it would be something like using the wrong GitLab ENV variable. We will use the workaround with the CI_JOB_ID instead of CI_PIPELINE_ID for now and are looking forward for a more native fix. Other than that our experience with Cypress has been amazing, it is a great product!

[damusnet]

Thanks @bahmutov!