Custom 'Should' error messages are not dynamic

Current behavior:

it('fails with custom message via overwrite of should', () => {
    cy.wrap(1).should('eq', 1, { message: 'custom error message 1'})
    cy.wrap(1).should('eq', 2, { message: 'custom error message 2'})
  })

Actual error message:

CustomError:  custom error message 1

Expected error Message:

CustomError:  custom error message 2

Desired behavior:

Custom error messages should be dynamic and not static.

If a ‘should’ fails, then the custom message of where a ‘should’ failed is expected to be displayed and not the first custom error message on the test run.

Steps to reproduce: (app code and test code)

  /*
    Ref: https://docs.cypress.io/api/cypress-api/custom-commands.html#Overwrite-Existing-Commands
    Add this to /cypress/support/commands.js
  */
  Cypress.Commands.overwrite('should', (originalFn, actual, assertion, expected, options) => {
    if (options && options.message) {
      cy.on('fail', (error, runnable) => {
        error.name = 'CustomError'
        error.message = options.message
        throw error // throw error to have test still fail
      })
    }
    return originalFn(actual, assertion, expected, options)
  })

it('fails with custom message via overwrite of should', () => {
    cy.wrap(1).should('eq', 1, { message: 'custom error message 1'})
    cy.wrap(1).should('eq', 2, { message: 'custom error message 2'})
  })

Versions

v3.8.0

OS Windows 10, browser chrome

[sainthkh]

@jennifer-shehane It has nothing to do with #3171. Reason below:

@dizhar According to Node.js EventEmitter documentation, on() function calls all event listeners in the ordered of registration. (Internally, cy object inherits methods in node.js EventEmitter).

In your case, you registered your first listener with {message: 'custom error message 1'} with the first should command. And the second should fails a lot of times and it adds a lot of listeners.

Then, when the test fails, it tries to run all the listeners. Then, when it runs the first listener registered by the first should. It throws the first error object (with "custom error message 1".

That’s why it threw the first error message.

Then, how should we solve this problem?

The simplest method is to remove 'fail' listeners before registering new ones like below:

Cypress.Commands.overwrite('should', (originalFn, actual, assertion, expected, options) => {
  if (options && options.message) {
    cy.removeAllListeners('fail') // remove all 'fail' listeners
    // And we're starting from fresh!
    cy.on('fail', (error, runnable) => {
      error.name = 'CustomError'
      error.message = options.message
      throw error // throw error to have test still fail
    })
  }

  return originalFn(actual, assertion, expected, options)
})

But if you created multiple custom commands and they use 'fail' event, they can cause flaky tests. In that case, you need to remove listeners when the should ran successfully like below:

Cypress.Commands.overwrite('should', (originalFn, actual, assertion, expected, options) => {
  if (options && options.message) {
    const listener = (error, runnable) => {
      error.name = 'CustomError'
      error.message = options.message
      throw error // throw error to have test still fail
    }

    const removeListener = () => {
      cy.removeListener('fail', listener)
      cy.removeListener('command:end', removeListener)
    }

    cy.on('fail', listener)
    cy.on('command:end', removeListener)
  }

  return originalFn(actual, assertion, expected, options)
})

Note: "command:end" event is fired when command is ended. For more information, please check here.

I hope this solved your problem.

[dizhar]

That’s great news! I am looking forward to it.

Also thank you for the quick response.

Best Regards’ Daniel Izhar

On Thu, Jan 9, 2020, 8:16 AM Kukhyeon Heo [e-mail] wrote:

@dizhar https://github.com/dizhar #6066 https://github.com/cypress-io/cypress/pull/6066 is merged and everything will be fixed in the next version of Cypress.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cypress-io/cypress/issues/6060?email_source=notifications&email_token=ABTIQA5X3FMTVO73TOQ4HV3Q426LBA5CNFSM4KAJMUPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIPD6GQ#issuecomment-572407578, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTIQA3AKTH66CATLU7EVETQ426LBANCNFSM4KAJMUPA .