Global app + spec error handling is inconsistent

Error Handling

Cypress does not consistently handle app/spec code global error handlers consistently.

Currently Cypress overrides autWindow.onerror and fails the test, but gives you an event listener hook cy.on('uncaught:exception', ...) to allow you to turn this behavior off.

• https://docs.cypress.io/api/events/catalog-of-events.html#Uncaught-Exceptions
• https://docs.cypress.io/api/events/catalog-of-events.html#App-Events

Observations (problems and inconsistencies)

• spec code onerror fails the test
• app code onerror fails the test
• spec code onunhandledrejection does not fail the test
• app code onunhandledrejection does not fail the test
• setting autWindow.addEventListener('error', ...) still fails the test
• setting autWindow.onerror = ... overrides Cypress, does not fail the test
• <script src="http://does.not.exist/resource"><script> does not fail the test
• <img src="http://does.not.exist/resource" /> does not fail the test
• app / spec uncaught errors are not logged
• app / spec unhandled rejections are not logged

Debates

• is it intentional not to fail the test if the app code has overridden Cypress’s global autWindow.onerror?
• if its NOT intentional, this is obviously a breaking change and would likely cause people to have to utilize cy.on('uncaught:exception', () => false) not to fail their tests
• should we add onunhandledrejection handling to behave like onerror ?
• should we fail the test when <img /> or <script /> tags have their onerror handler called?
• should we fail the test even if window.onerror is set by app code UNLESS the default behavior is preventedDefault?
• if we add onunhandledrejection should this have its own event listener such as cy.on('unhandled:rejection') or should we expand the uncaught:exception event or arguments to specify which type it is: cy.on('uncaught:exception', (error, errorType, runnable) => {})

Notes

• Apparently I first thought of this back in the good ol’ days of 2016: https://github.com/cypress-io/cypress/issues/243

Recommendations

• If no handlers are defined, fail the test
• If handlers are defined, don’t fail the test, but still fire uncaught:exception to allow the user to still fail the test
• Add event handlers for onunhandledrejection
• Add event handlers for resources that fail would have their onerror handler called, but apply the same rules above
• Always log out the events via Cypress.log() regardless whenever these events occur to help specify their origin

[Andarist]

is it intentional not to fail the test if the app code has overridden Cypress’s global autWindow.onerror?

I don’t think this should be counted as intentional. It seems to be just a byproduct of Cypress assigning its error handler using direct assignment to the autWindow.onerror property rather than using autWindow.addEventListener API. I’ve actually implemented a change to this in this PR: https://github.com/cypress-io/cypress/pull/9077 . It could be merged into v6 if you agree with the approach.

should we add onunhandledrejection handling to behave like onerror ?

I believe that is very reasonable and expected. As a user, I was quite surprised that this isn’t already consistent. I don’ quite care how and where the uncaught error was thrown - the important part is that I should fix it and making onunhandledrejection fail tests would help me to do that. I’ve started working on that here: https://github.com/cypress-io/cypress/pull/9118

should we fail the test when or <script /> tags have their onerror handler called?

This is a tricky one because it’s hard to determine if the error was caught by the user or not. If you could detect that (by monkey-patching prototypes maybe?) then uncaught errors could fail tests - especially for scripts. Images are a little bit more arguable but the error is still an error, so maybe worth failing in that case as well?

if we add onunhandledrejection should this have its own event listener such as cy.on(‘unhandled:rejection’) or should we expand the uncaught:exception event or arguments to specify which type it is: cy.on(‘uncaught:exception’, (error, errorType, runnable) => {})

The same error handler would IMHO be preferred. I also don’t think it should really matter what’s the origin of the error - but I guess providing an extra meta param won’t hurt anyone. I don’t know though how one would really utilize that. Maybe roll without it and see if people need it?

---

As a user, I might have attached .onerror handler for a variety of reasons but I don’t think this is a strong signal to turn off Cypress’ default behavior, such a behavior change would catch me completely off guard.

[brian-mann]

@JessicaSachs

I think we should warn the user, but pass the test

We could do that by adding the Cypress.log() events - which would serve as a warning to the user. But possibly we could still fire the uncaught:exception handler, but treat the return false as the default, but accept a return true which does the inverse and fails the test.

Examples:

Current Implementation

// index.html
<script>
foo.bar()
</script>

cy.visit('/index.html') // fails test

// index.html
<script>
foo.bar()
</script>

cy.on('uncaught:exception', (err) => {
  return false // prevents failing test
})
cy.visit('/index.html') // does not fail test

// index.html
<script>
// override global onerror handling
window.onerror = () => {}

foo.bar()
</script>

cy.on('uncaught:exception', (err) => {
  // this is not hit
})
cy.visit('/index.html') // does not fail test

Possible Implementation

Option 1. Still fail the test

// index.html
<script>
// override global onerror handling
window.onerror = () => {}

foo.bar()
</script>

cy.on('uncaught:exception', (err) => {
  // this is now hit and fails the test
  // unless the user returns false here
})
cy.visit('/index.html') // fails test

Option 2. Don’t fail test by default unless return true

// index.html
<script>
// override global onerror handling
window.onerror = () => {}

foo.bar()
</script>

cy.on('uncaught:exception', (err) => {
  // this is now hit but since the event was
  // preventedDefault, we could NOT fail the test
  // unless the user returns true (the inverse)
  return true
})
cy.visit('/index.html') // fails test