Total assertion count expectations are inaccurate (with async expects)

🐛 Bug Report

expect.hasAssertions() and expect.assertions() incorrectly count assertions as part of a test when they should not be.

To Reproduce

The most basic example I could create to demonstrate

test('hasAssertions should fail expects in promises', () => {
  expect.hasAssertions();
  // Note that the expect below does not "count" as a failed assertion for this
  // test, but hasAssertions() also does not fail as it should!
  Promise.resolve().then(() => expect(true).toBe(false));
});

Expected behavior

The test above should fail, the async assertion should be ignored towards the assertion count. The promise chain, where the expect is executing is not part of the test. We can tell that because the test is not marked as failing even with a blatantly broken assertion. When the expect runs the test is “complete” and should not be counted as part of assertionCount in Jest.

Link to repl or repo (highly encouraged)

I forked the repo and wrote an integration test to demonstrate the problem. Branch

Permalink to the spec itself

https://repl.it/repls/DramaticCuteAssignments

Run npx envinfo --preset jest

❯ npx envinfo --preset jest

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (4) x64 Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
  Binaries:
    Node: 8.11.1 - ~/.nvm/versions/node/v8.11.1/bin/node
    Yarn: 1.15.0 - /usr/local/bin/yarn
    npm: 5.6.0 - ~/.nvm/versions/node/v8.11.1/bin/npm

Notes

I do realize that the test is bad, it’s not very well written. But tests like these do happen IRL and assertion count matchers is what you would want to use to catch them. Currently they do not work for this scenario.

This was pretty interesting and I dug into it a bunch, trying to see if there is an easy fix. But I was unable to locate an obvious mistake. It seems like the assertionCount logic (expect) is separate from the logic marking tests as failures (jest-jasmine2?) so there isn’t a simple way to marry the two.

[ballercat]

Ooh, the zone.js library is very cool.

How I stumbled upon this is trying to write a “plugin” for jest which would catch these type of runtime issues. I have incorporated zone.js library into it as a PoC, and it seems to be working rather well. I’m going to throw this at a few thousand tests I have available and report back. Thanks!

[ballercat]

@SimenB Thanks for the response and I agree 100% RE: the problem of global expect not being bound to a single test.

It’s probably a bit too late in the lifecycle of Jest to expand/change it’s API in such a way but I for one would love to see an addition to the test(string, fn) API with something like test.strict that would directly provide the expect Fn as a variable instead of relying on the global. Similar ideas have worked for other test runners (ava comes to mind).

Ex:

test.strict('this test will fail assertions post test completion', (expect) => {
   setTimeout(() => expect(true).toBe(true), 10000);
});