Any unhandled error causes acceptance tests to fail, making it impossible to test error states

Hi!

In our app, we catch errors in order to display flash messages to the user, letting them be aware of an error and offering a chance to retry.

We choose to re-throw the error after displaying the flash message. This is because we want the error to be caught by an error tracking service such as Sentry, Bugsnag, TrackJS, Raygun, etc.

The problem is that QUnit tests would crash on any uncaught error, making it impossible to test error reporting.

There are two userland workarounds from this issue:

1. Choose not to re-throw an error. I don’t like this approach because it would require adding manual error tracking commands into every catch block.
2. Re-throwing conditionally, only in non-test environment. This is more or less OK, but I don’t feel comfortable adding exceptions for tests. I believe, it’s a slippery slope.

Previously, it was possible to override Ember.Test.adapter.exception and prevent tests from failing on uncaught assertions. The best part is that you could check for a condition (e. g. a certain error code or message) and re-throw an error if it was not the one you were expecting.

But according to ember-qunit source, it is no longer possible in modern Ember:

// Ember 2.17 and higher do not require the test adapter to have an `exception`
// method When `exception` is not present, the unhandled rejection is
// automatically re-thrown and will therefore hit QUnit's own global error
// handler (therefore appropriately causing test failure)

How do I prevent QUnit from crashing on unhandled errors? What is the modern replacement for overriding Ember.Test.adapter.exception?

[rwjblue]

@lolmaus - I’ll try to reply to the various sections independently (as there are a few separate points you are making).

The problem is that QUnit chooses to crash tests on all unhandled promise rejections (with exceptions mentioned above).

FWIW, I think this is the correct stance for QUnit to take (and IIRC I implemented it waaaaaaay back in 2017 https://github.com/qunitjs/qunit/pull/1241). Unhandled exceptions (and unhandled promise rejections) are bad, and in most circumstances point to an actual problem in the application.

If the case of failed tests on unhandled native promises “works as intended”, then we need guidelines how to deal with unhandled promises

I totally agree with you here! We absolutely should document how to handle situations like @amk221’s.

It’s not that obvious, because some async methods are hooks which are handled by Ember internals (sometimes too aggressively — you might not see the error at all, or it might be caught and rethrown elsewhere with an entirely irrelevant stack trace), but others like an action in @amk221’s example are not handled automatically.

I agree that it is unclear how Ember’s internals handle unhandled rejections. I think we should update the documentation for all async route hooks to explicitly say what happens when an exception occurs. For example, if an exception occurs during model and you have an error substate or error action that rejection is automatically handled for you (this is why you don’t commonly see the problem you are reporting for that category of code). I also think that doing that documentation will significantly help folks think about how exceptions / rejections are handled by the system (and therefore how they should be handled by their own code).

A typical Ember developer does not have a habit of catching promises because Ember guides never state that they should, and it works in most cases until it suddenly doesn’t, and it’s absolutely frustrating.

To reiterate, I agree with you. We should increase both API and guide documentation for exceptions/rejections and how application developers should handle those situations. FWIW, I do not think the answer is “never leave them uncaught” (as sometimes you really do intend for an error / rejection to bubble all the way up loudly).

How do we even handle promises? Take @amk221’s example. What can be done to resolve the problem? The only thing I can think of is replacing throw error with something like this.metrics.trackError(error). Is that what we’re supposed to do?

I am totally on board with the idea that we should identify specific use case / scenarios, and solve for them. That will help us know what to document (and where). Unfortunately, the example doesn’t have enough information. What is triggering this action? What does the test look like? What kind of test? etc.

---

Ultimately, I think we should (mostly in order):

• attempt to summarize the cases that we are trying to work through (no, the general “unhandled rejections” bucket is not good enough 😝 )
• ensure that they are all possible to test
• determine what changes are needed (either here in ember-qunit, in @ember/test-helpers, ember-source, etc) to make those tests a tad less onerous (IMHO the tests making it clear that an unhandled rejection will be happening is totally fine)
• once we understand the scenarios, create detailed issues (or PRs 😉) for the API docs and guides that we think we need

[buschtoens]

I’ve started a PR to finally fix this: ember-test-helpers#1194. I would appreciate your thoughts!