Better timeout errors through deadline checking

🚀 Feature Proposal

1. Expose the computed test timeout to tests, so they can make decisions based on this timeout.

2. Provide a utility to check if the timeout is expired, or nearly expired, so that tests can unwind, and produce a better error.

3. Use this utility wherever possible, such as during expect or await.

I have a monkey-patching/internals-abusing implementation here: https://github.com/FauxFaux/jest-fixup-timeouts#jest-fixup-timeouts . In this implementation, I have gone with the proposal below. There’s no need for any of this to be monkey patching, it feels like it would fit quite well as a PR, but it’s way easier to prototype and test this way.

1. I propose calling the timeout “test.deadline”, and it being a unix timestamp (or Date?). A deadline is Date.now() + timeout - fudge, for some fudge which allows realistic code to unwind (20ms? 1%?).

2. I have added a function named expect.withinDeadline which takes a Promise, and races it against test.deadline, resolving with the result, or rejecting with the result, or rejecting with a TimeoutError. It would also be great if await expect(..).resolves did this, but I failed to monkey-patch it.

3. There’s an (optional) babel plugin which rewrites (await foo()) to (await expect.withinDeadline(foo()) so you get this on all awaits, instead of just the ones covered by expect. This can be applied to tests, or wherever you want.

Motivation

We use jest-circus essentially as an integration testing framework. Many of our tests fail with “timeout exceeded, consider increasing the timeout”, with no indication of what was happening at the time. We are then informed that our test leaked handles (as it is still running). This is a very poor user experience.

Example

Consider a test like:

it('removes existing resources', async () => {
  await registerResources('yellow');
  await expect(removeResources('yellow')).resolves.toBe('204 gone');
});

We have this fail with:

Error: thrown: "Exceeded timeout of 5000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

This gives you no indication of which operation failed; what the test was doing; which bit is causing this flaky behaviour.

With the ha… proposal, this will give an error more like:

  ✕ removes existing resources

    deadline exceeded (waited here for 4932ms)

      2 | it('removes existing resources', async () => {
    > 3 |    await registerResources('yellow');
        |    ^
      at Object.<anonymous> (test/demo.spec.ts:3:3)

Pitch

Three parts; the first two feel very related to the core; as they’re a new major part of the “globals” (test/expect) API.

1. Deadlines: It’s a piece of data we hand to the core platform, but which it then hides from us. We could stash it in an environment variable, or another global, maybe? Feels very much like core to me.

2. expect.withinTimeout doesn’t need to be core, if test.deadline exists. However:

• there’s probably very limited uses of test.deadline outside of that utility function (the only one I’ve come up with is to pass it to the RPC client, so it can pass it on to remote services, and they can do something sensible),
• and it’s not a matcher, *and extending the expect object directly isn’t normal (or possible?), and it doesn’t feel like it should be elsewhere.

1. The babel plugin is fine as a 3rd party plugin. But, the feature isn’t very useful without it, and the plugin is quite simple, by babel standards; maybe it can just be merged into babel-jest?

[akauppi]

I’m interested in this.

My main problem with Jest 27 has become tests timing out (which may lead to the Jest process never finishing to OS level, but that’s another plot). Had made some designs of my own (no PR) before finding this.

1. Expose the computed test timeout to tests, so they can make decisions based on this timeout.

This allows making work-arounds, but doesn’t cure the deeper problems.

1. Provide a utility to check if the timeout is expired, or nearly expired, so that tests can unwind, and produce a better error.

This is what I’m mainly after.

One of the related problems is that JavaScript promises are not cancellable - but they can be cancelled ”from within” if one is in charge of the original new Promise code. For my use cases, having a beforeTimeout callback would be sufficient. The Promise body could get that knowledge, and turn itself to either resolve (sometimes in my code ”timed out” is the expected behaviour) or reject.

Why this matters is that it also provides a mechanism to make sure tests don’t leave Promises dangling. At the moment, this happens, and at least for my CI, it means the Jest process does not return to the OS and the CI also times out.

1. Use this utility wherever possible, such as during expect or await.

This is where the approaches differ - and where discussion would be welcome. I would not touch expect or await but handle the issue deeper, at the source of the Promise waited upon itself.

My offer: to take part in design/testing of such a mechanism, for making the Jest user experience better.

[jeysal]

I think in the end resolves/rejects will not be sufficient for this to make an impact. Because of async-await, I don’t find myself using them a lot anymore, mostly just for rejects. expect(await findByText('...')).toBeVisible() is just so much prettier than await expect(findByText('...')).resolves.toBeVisible(). So I think including it just for resolves / rejects is almost the same as not including it by default at all and pointing people to a third party add-on, meaning it’ll end up with little usage. I think if we are convinced by the solution, think it brings a lot of benefit and has no pitfalls / ugly edge cases, then we want to go all in on it.