TestCafe returns 0 exit code on test failure

What are you trying to achieve?

Using the TestCafe helper, have CodeceptJS return non-zero exit codes when tests fail.

What do you get instead?

Zero exit code when tests fail. This isn’t an issue if we switch to e.g. Puppeteer.

Note that the below example is on Windows/Powershell because it’s my local dev environment but our real issue is getting our tests working in Travis. The same problem happens there.

Provide console output if related. Use --verbose mode for more details.

PS C:\Users\jeremy\Documents\projects\test-codeceptjs> npx codeceptjs run --verbose
CodeceptJS v2.3.6
Using test root "C:\Users\jeremy\Documents\projects\test-codeceptjs"
Helpers: TestCafe
Plugins: screenshotOnFail

Test --
    [1] Starting recording promises
    Emitted | suite.before ([object Object])
  Test
    Emitted | test.before ([object Object])
 » [_before] Starting testcafe browser...
 Running tests in:
 - Chrome 79.0.3945.79 / Windows 10
    Emitted | test.start ([object Object])
    Emitted | step.before (I am on page "https://www.google.com")
    Emitted | step.after (I am on page "https://www.google.com")
    Emitted | step.before (I see "Nothing")
    Emitted | step.after (I see "Nothing")
    Emitted | step.start (I am on page "https://www.google.com")
    I am on page "https://www.google.com"
    Emitted | step.passed (I am on page "https://www.google.com")
    Emitted | step.finish (I am on page "https://www.google.com")
    Emitted | step.start (I see "Nothing")
    I see "Nothing"
    [1] Error | Error: AssertionError: No element with text "Nothing" found: expected 0 to be above 0
    Emitted | step.failed (I see "Nothing")
    Emitted | step.finish (I see "Nothing")
    [1] Error | Error: AssertionError: No element with text "Nothing" found: expected 0 to be above 0
    [1] Starting <teardown> session
    Emitted | test.failed ([object Object])
    Emitted | test.finish ([object Object])
  × FAILED in 6189ms

    [2] Starting recording promises
    Emitted | test.after ([object Object])
 » [_after] Stopping testcafe browser...
    Emitted | suite.after ([object Object])

-- FAILURES:

  1) Test
       Test:
     AssertionError: No element with text "Nothing" found: expected 0 to be above 0

  Scenario Steps:

  - I.see("Nothing") at Test.<anonymous> (src\test.spec.js:7:7)
  - I.amOnPage("https://www.google.com") at Test.<anonymous> (src\test.spec.js:6:7)

  Error: AssertionError: No element with text "Nothing" found: expected 0 to be above 0
      at mapError (node_modules\codeceptjs\lib\helper\testcafe\testcafe-utils.js:29:11)
      at invokeCallback (node_modules\pinkie\index.js:60:12)
      at Array.forEach (<anonymous>)
      at publish (node_modules\pinkie\index.js:147:32)
      at Array.publishRejection (node_modules\pinkie\index.js:157:2)
      at Immediate.asyncFlush (node_modules\pinkie\index.js:17:19)
      at processImmediate (internal/timers.js:441:21)
      at process.topLevelDomainCallback (domain.js:130:23)


  FAIL  | 0 passed, 1 failed   // 14s
    Emitted | global.result ([object Object])
    Emitted | global.after ([object Object])
PS C:\Users\jeremy\Documents\projects\test-codeceptjs> $?
True // Should be False

Provide test source code if related

Feature('Test');

Scenario('Test', async (I) => {
    I.amOnPage('https://www.google.com');
    I.see('Nothing');
});

Details

• CodeceptJS version: 2.3.6
• NodeJS Version: 12.13.1
• Operating System: Windows 10/Ubuntu Trusty 14.04 (Travis)
• TestCafe: 1.7.0
• Configuration file:

exports.config = {
  tests: './src/*.spec.js',
  output: './output',
  helpers: {
    TestCafe : {
      browser: process.profile || 'chrome',
      show: false
    }
  },
  include: {},
  bootstrap: null,
  mocha: {},
  name: 'test-codeceptjs',
  timeout: 4000,
  plugins: {
    screenshotOnFail: {
      enabled: true
    }
 }
}

[KMKoushik]

You can put it inside https://github.com/Codeception/CodeceptJS/blob/master/lib/listener/exit.js as well. I hope it will work

Yes this will work. I will try once and let you know

[DavertMik]

Ok, so maybe we could introduce our own beforeExit callback which will do process.exit(1) whenthere are failures? You can put it inside https://github.com/Codeception/CodeceptJS/blob/master/lib/listener/exit.js as well. I hope it will work