Weird flicker in video for end to end test on updating to v10.10.0

Current behavior

We were previously using cypress version 9.7.0 and now want to update to version 10.10.0. However, we sometimes get an issue with the end-to-end tests when running on GitLab CI.

As you can see from the video, a weird flicker happens during the test and it eventually fails. Does it seem that the test is jumping up and down the commands as it was running two instances of the test? The video is only showing the start of the test and is pixelated since this is a private project.

https://user-images.githubusercontent.com/43796105/197748959-0059a24d-fa4a-4f33-95e0-5579ec479ccd.mp4

We haven’t been able to reproduce this behavior locally, and it doesn’t constantly happen on the CI. It is also not specific to this test, sometimes the other end-to-end tests fail occasionally and the videos have the same weird flicker. These tests didn’t have this issue before upgrading cypress version.

Desired behavior

We would like to be able to update cypress to version 10, but this issue is delaying the update since we cannot rely on the end-to-end tests for our CI.

Test code to reproduce

We can’t provide the full test code since this is a private project.

• cypress.config.ts

import { defineConfig } from "cypress";

export default defineConfig({
    chromeWebSecurity: false,
    defaultCommandTimeout: 15000,
    viewportWidth: 1920,
    viewportHeight: 1080,
    videoUploadOnPasses: false,

    projectId: "xxxxxx",

    e2e: {
        setupNodeEvents(on, _config) {},

        retries: {
            runMode: 0,
        },

        specPattern: "cypress/e2e/**/*.{ts,tsx}",
    },
});

• cypress-dockerfile

FROM cypress/included:10.10.0

RUN npm install -g pnpm@6.2.1

# other packages installed necessary to run our e2e tests

• gitlab-ci.yml

itest:end-to-end:
    image: ${ITEST_IMAGE}
    stage: itest
    script:
      # backend necessary scripts
  
      - cd frontend
      - echo -e "\e[0Ksection_start:`date +%s`:frontend-deps[collapsed=true]\r\e[0KInstalling frontend deps"
      - CYPRESS_INSTALL_BINARY=0 pnpm install --no-verify-store-integrity --frozen-lockfile -r
      - echo -e "\e[0Ksection_end:`date +%s`:frontend-deps\r\e[0K"
      - cypress run
        --config baseUrl="http://localhost:8001/",defaultCommandTimeout=60000,watchForFileChanges=false,pageLoadTimeout=100000,numTestsKeptInMemory=0
        --browser chrome
        --reporter mocha-junit-reporter
        --reporter-options 'mochaFile=cypress/reports/junit/junit-[hash].xml'
        --env KC_HOSTNAME="localhost:8080"
        --spec clusters-active.ts
    artifacts:
      when: on_failure
      paths:
        - frontend/cypress/screenshots
        - frontend/cypress/videos
        - logs
      expire_in: 15 days
      reports:
        junit: frontend/cypress/reports/junit/*.xml

• gitlab output

$ cypress run --config baseUrl="http://localhost:8001/",defaultCommandTimeout=60000,watchForFileChanges=false,pageLoadTimeout=100000,numTestsKeptInMemory=0 --browser chrome --reporter mocha-junit-reporter --reporter-options 'mochaFile=cypress/reports/junit/junit-[hash].xml' --env KC_HOSTNAME="localhost:8080" --spec clusters-active.ts
[10659:1017/234749.919499:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[10659:1017/234750.894445:ERROR:zygote_host_impl_linux.cc(263)] Failed to adjust OOM score of renderer with pid 10889: Permission denied (13)
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[10889:1017/234750.909020:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
====================================================================================================
  (Run Starting)
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        10.10.0                                                                        │
  │ Browser:        Chrome 100 (headless)                                                          │
  │ Node Version:   v16.14.2 (/usr/local/bin/node)                                                 │
  │ Specs:          1 found (clusters-active.ts)                                                   │
  │ Searched:       cypress/e2e/customer/clusters-active.ts                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  clusters-active.ts                                                              (1 of 1)
Timed out waiting for the browser to connect. Retrying...
  (Results)
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        9                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      8                                                                                │
  │ Screenshots:  2                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 minute, 40 seconds                                                             │
  │ Spec Ran:     clusters-active.ts                                                               │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
  (Screenshots)
  -  /builds/.../frontend/cypress/screenshots/clusters-act     (1280x720)
     ive.ts/test -- before all hook (failed).png                        
  -  /builds/.../frontend/cypress/screenshots/clusters-act     (1280x720)
     ive.ts/test -- after all hook (failed).png                         
  (Video)
  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /builds/.../frontend/cypress/vid    (6 seconds)
                          eos/clusters-active.ts.mp4                                                
====================================================================================================
  (Run Finished)
       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  clusters-active.ts                       01:40        9        -        1        -        8 │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 1 failed (100%)                     01:40        9        -        1        -        8  

Cypress Version

10.10.0

Node version

v16.14.2

Operating System

cypress/included:10.10.0

Debug Logs

No response

Other

Which debug logs would be more relevant for this? DEBUG=cypress:* has too many logs, so the GitLab job exceeds the maximum logs saved before the test finishes.

[mschile]

Thanks for the update @lidiagc! I would love to narrow in on which Cypress version caused the regression. My initial guess is Cypress 10.8.0 may have broke it. Would you be able to try 10.7.0 and 10.8.0?

[mjhenkes]

@lidiagc, given that you’re seeing this less frequently on 11.0.1 would you be ok with us closing this issue?