run-ct returns with exit code 0 when tests fail - breaking CI detection of failed tests

Current behavior

This is a pretty big bug. All CI jobs running with run-ct will report success even if tests fail due to this bug.

In this screenshot, you are seeing the output of yarn cypress:component:cli --spec=packages/SearchInput/src/component_tests/ClearButton.test.tsx which runs this command:

    "cypress:component:cli": "cypress run-ct --project integration-tests",

Note that at the bottom, we see Done in 69.93s but no error. This Done is reported by yarn. When a test fails, it should look more like this (I added expect(false).toBe(true) to a jest test to simulate a failure):

cypress.json:

{
  "baseUrl": "http://localhost:6009",
  "retries": {
    "runMode": 2,
    "openMode": 0
  },
  "integrationFolder": "../packages",
  "testFiles": "**/integration_tests/*.test.tsx",
  "component": {
    "componentFolder": "../packages",
    "testFiles": "**/component_tests/*.test.tsx"
  }
}

Desired behavior

when a test fails, the shell error code should be non-zero. cypress run does this correctly.

Test code to reproduce

1. write a failing component test
2. run it with cypress run-ct
3. watch to see that the exit code is 0

Cypress Version

8.3.0

Other

tested with 7.6.0 as well, upgraded to be sure it was not already fixed.

[bahmutov]

I wonder if somewhere in the release and test pipeline a tool like https://github.com/bahmutov/cypress-expect could be used to verify the failing tests and the exit code

[cellog]

excellent, thank you, I’ll report back shortly, probably a few hours once these meetings are dun