[Feature] Define different exit codes to make CI/CD scripting easier

It looks like process.exit(1); is used universal throughout the project when something does not go as expected. process.exit(1); is also used when everything does execute properly, but not all test cases passed. Finally, it seems there are a couple cases were exit code 0 is associated with an error.

It would be very handy for CI/CD scripting to easily determine Playwright’s exit reason. For very basic support, it could be:

• 0: Everything executed properly with all tests passing
• 1: Everything executed properly, but with failing tests
• 2: Something unexpected happened during execution

For my needs, these three different exit code would be good enough. I would like to run tests in the pipeline - and I have some tests that will periodically fail due to external reasons. As long as all tests are running, I would like the pipeline to be successful - or perhaps marked as unstable. On the other hand, if something new was introduced that is causing the tests to break - then that is a problem to halt the pipeline over.

Depending on how granular you want to go, you could add even more exit codes. This is what pytest does for their exit codes:

• 0: All tests were collected and passed successfully
• 1: Tests were collected and run but some of the tests failed
• 2: Test execution was interrupted by the user
• 3: Internal error happened while executing tests
• 4: pytest command line usage error
• 5: No tests were collected

I’m not opposed to trying to add support for more consistent exit codes myself, but I want to make sure everyone is aligned on expected behavior before any time is spent on it.

Thanks!

[vitalics]

@lightswitch05 I understand your point. Not sure about possibilities for CLI exit codes, since you can only understand - if your file executes with/without errors. If you run the simple file with a playwright - you should take care of your own exit codes like in the example below

async function main(){
  try {
    // run playwight
  }  catch (err) {
    if(err === 'external reason') {
      process.exit(<my exit code>);
    }
  }
}

main();

I suppose each testing implementation should provide its exit codes. if we talking about python - PyTest should return their exit codes.

I would like to ask @mxschmitt to transform your request for the @playwright/test package if it’s a relevant feature request - since @playwright/test is the official test runner for playwright.

Addition: jest(@playwright/test based on jest) supports only exit code 1 if at least one test has been failed by expectation failing. I do research on exit codes in nodejs and the best way - do not to break backward compatibility. But in the case of a new implementation test runner - I suggest to use exit codes > 14. E.g.

• 0 - all tests passed
• 1 - all tests failed
• 14 - some tests failed
• 15 - cancel by user
• 7 - internal error happened while executing tests
• 16 - command line error? (or just exit code 1)
• 17 - No tests have been executed

[vitalics]

@joeflan for your purpose I would like to recommend wrap using script and capture error using bash/cmd.

Example

npm run test || exit 0