failing beforeAll() causes even passing tests in the scope to fail

🐛 Bug Report

I’m looking for the best practice for aborting a describe block if the test setup fails. For example, when testing an API, if authenticating fails, it’s pointless to run any other tests.

As suggested in the issue template, I searched StackOverflow first, where I found this incomplete answer, advising to place the initialization code in a beforeAll block. Hence, raising the issue here because

1. I haven’t seen anything documenting the practice of returning failure from beforeAll
2. If a beforeAll block fails, tests in that describe block are still run, and they fail, even if otherwise they would pass!

To Reproduce

describe('test that a 3rd party API remains consistent', () => {
  beforeAll(() => expect('login').toBe('successful'));  // this will fail
  test('API function 1', () => expect(1).toBe(1));  // each...
  test('API function 2', () => expect(2).toBe(2));  // ...of these...
  test('API function 3', () => expect(3).toBe(3));  // ...will fail too
});

Expected behavior

Jest should report that the beforeAll() failed, and bail the describe scope without executing further tests in it. If it’s somehow intended behavior that all tests should still be executed, and still marked as failed (which I find odd), this should be documented under beforeAll and in the Setup and Teardown guide. Other tests in the file, outside of the failing block, should still be executed.

Link to repl or repo (highly encouraged)

https://repl.it/@DanDascalescu/beforeAll-failure-should-bail-the-test

Run npx envinfo --preset jest

  System:
    OS: Linux 4.15 Ubuntu 16.04.4 LTS (Xenial Xerus)
    CPU: x64 Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
  Binaries:
    Node: 10.5.0 - /usr/local/bin/node
    Yarn: 1.7.0 - /usr/bin/yarn
    npm: 6.2.0 - /usr/local/bin/npm
  npmPackages:
    jest: ^23.4.1 => 23.4.1 

See also

#6527

[SimenB]

To my eyes this just looks like an issue with the reporting. beforeAll is only run once and none of the other tests actually execute, but they all fail with the same reason, and they all print the same error.

We should probably avoid printing the individual tests if beforeAll fails. Right now the implementation just checks if we’ve had an error and if yes, print it and fail the test without actually executing it

[chrismwendt]

jest-circus seems to not fix this. All 3 tests still run and fail.

Here’s the command I ran:

$ yarn run jest --testRunner=jest-circus/runner --bail

Am I missing a configuration value somewhere?