Error in handleTestEvent fails the entire test suite (no tests marked as being run)

🐛 Bug Report

If an error happens inside a handleTestEvent(event), the entire suite fails to run, even though the error only happens for one of the test cases.

To Reproduce

Place these files in a server/ directory and run as jest server/

jest.config.js

module.exports = {
  projects: [
    {
      rootDir: 'server/',
      testEnvironment: '<rootDir>/CustomEnvironment.js',
      testRunner: 'jest-circus/runner',
    },
  ],
};

CustomEnvironment.js

const NodeEnvironment = require('jest-environment-node');

class CustomEnvironment extends NodeEnvironment {
  handleTestEvent(event) {
    if (event.name === 'test_done' && event.test.errors.length > 0) {
      throw new Error('this will mark the entire suite as failed');
    }
  }
}

module.exports = CustomEnvironment;

foo.test.js

it('pass', () => {
  expect(true).toBeTruthy();
});

it('fail', () => {
  expect(false).toBeTruthy();
});

This gives the following output:

> jest server/
 FAIL  server/foo.test.js
  ● Test suite failed to run

    this will mark the entire suite as failed

       6 |   handleTestEvent(event) {
       7 |     if (event.name === 'test_done' && event.test.errors.length > 0) {
    >  8 |       throw new Error('this will mark the entire suite as failed');
         |             ^
       9 |     }
      10 |   }
      11 | }

      at CustomEnvironment.handleTestEvent (CustomEnvironment.js:8:13)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        1.042s
Ran all test suites matching /server\//i.

Notice that tests say Tests: 0 total.

Expected behavior

I expect each individual test case that fails in handleTestEvent to be reported as failed in handleTestEvent, but not for all other test cases passing to fail. They should still be reported as having being run successfully.

I expect the tests to say Tests: 1 failed, 1 passed, 2 total, and that maybe the main output for the failure should say ✕ fail (2ms) (failed in handleTestEvent) or similar.

Link to repl or repo (highly encouraged)

envinfo

  System:
    OS: Linux 4.4 Ubuntu 16.04.5 LTS (Xenial Xerus)
    CPU: (4) x64 Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
  Binaries:
    Node: 12.13.0 - ~/.config/nvm/12.13.0/bin/node
    npm: 6.12.0 - ~/.config/nvm/12.13.0/bin/npm
  npmPackages:
    jest: 25.1.0 => 25.1.0

[jeysal]

I think throwing is an internal error of the test runner (not a valid case of an implementation) and should therefore result in ‘Test suite failed to run’. I don’t think we need to distinguish between different events here, most of them would not have a better way of handling than ‘Test suite failed to run’ anyway.

[thernstig]

Agree that is a solution, and I am using that. I still felt it was weird for the entire suite to be marked as failed? So thought I’d submit this in case you also think this could be handled better, to help some other future user.