The exception object of an error that caused a step to fail is no longer accessible in custom formatters as of version 7

In version 4 of cucumber it was possible to access the exception which caused a step to fail in a custom formatter.

{
  "sourceLocation": {
    "uri": "features\\tests\\ecom\\family\\get-family-from-member.feature",
    "line": 7
  },
  "actionLocation": {
    "uri": "features\\tests\\ecom\\family\\steps_definitions\\get-family-from-member.js",
    "line": 8
  },
  "result": {
    "duration": 10065,
    "exception": {
		... Exception data ...
	}
  }
}

In version 7, out of the box there is only a “message” property which contain a serialized stacktrace but that’s it.

Here is an example test step object:

{
  "attachments": [],
  "keyword": "Given ",
  "result": {
    "status": "FAILED",
    "message": "Error: Request failed with status code 500\n    at createError (C:\\dev\\slp-api-consumers\\node_modules\\axios\\lib\\core\\createError.js:16:15)\n    at settle (C:\\dev\\slp-api-consumers\\node_modules\\axios\\lib\\core\\settle.js:17:12)\n    at C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:87:29\n    at new Promise (<anonymous>)\n    at $If_1 (C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:86:30)\n    at C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:97:18\n    at new Promise (<anonymous>)\n    at responseInterceptor (C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:21:10)\n    at C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\index.js:130:67\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)\n    at Object.exports.create (C:\\dev\\slp-api-consumers\\typescript-client\\src\\api\\slp\\test\\organization.api.ts:15:5)\n    at createPopulatedOrganismeOptions (C:\\dev\\slp-api-consumers\\typescript-client\\src\\common-procedures\\create-populated-organisme.ts:95:27)\n    at CustomWorld.<anonymous> (C:\\dev\\slp-api-consumers\\integration-tests\\step-definitions\\groups.steps.ts:50:25)",
    "duration": {
      "seconds": "10",
      "nanos": 73000000
    }
  },
  "actionLocation": {
    "uri": "step-definitions\\groups.steps.ts",
    "line": 41
  },
  "sourceLocation": {
    "uri": "features\\my-desk\\groups\\create-group-students.feature",
    "line": 27
  },
  "text": "a class with a teacher, 30 seats and 0 students exists"
}

We are currently porting our test suite from version 4 to version 7 and we had a custom formatter which displayed some information about our system based on request responses which we attached to the exception that our http client triggered upon failure. It was very useful to have access to the exception in the formatter for this reason.

Is there a different way to access exceptions now? Would you accept a pull request adding it to the result object of the test step object?

[jan-molak]

@charlierudolph - I agree with @oleduc, the formatted stack trace makes it very difficult for custom formatters to understand what error has been thrown and with what properties. It also makes it impossible to deserialise such an error to extract its details (i.e. the original expected and actual properties of an AssertionError), call methods on it, etc.

However, I’m not sure if adding a new details field on TestStepResult itself would help to resolve the issue. To my understanding, anything that’s passed via Cucumber messages needs to be serialised. Cucumber, of course, has no way of knowing how to serialise the errors so that they could be reliably de-serialised into valid objects afterwards.

So what I think could work here is to:

• return the original error from StepRunner.run method, alongside the TestStepResult:

// cucumber-js/src/runtime/step_runner.ts 
export async function run({
  defaultTimeout,
  hookParameter,
  step,
  stepDefinition,
  world,
}: IRunOptions): Promise<{ 
    result: messages.TestStepResult, 
    error: any,            // "any" because it could be Error but also anything else people might feel like throwing...
}>{ 
  // ...
}

• extend ITestStepHookParameter to include error, which would allow afterStep and after hooks to receive such error without breaking backwards compatibility:

export interface ITestStepHookParameter {
  gherkinDocument: messages.GherkinDocument
  pickle: messages.Pickle
  result: messages.TestStepResult
  error?: any
  testCaseStartedId: string
  testStepId: string
}

• do the same with ITestCaseHookParameter:

export interface ITestCaseHookParameter {
  gherkinDocument: messages.GherkinDocument
  pickle: messages.Pickle
  result?: messages.TestStepResult
  error?: any
  testCaseStartedId: string
}

I think this approach would:

• allow custom formatters to have access to the original error object
• maintain backwards compatibility
• avoid polluting Cucumber messages with duplicate information

@aslakhellesoy, @charlierudolph, @davidjgoss - Please let me know if you’re happy with the approach and I can prepare a PR

[oleduc]

That said, we’re encouraging our community to build formatters around the standardised message protocol, which allows formatters to be used by all cucumber implementations. This api is also more stable.

Here is an example : https://github.com/cucumber/cucumber-js/blob/master/src/formatter/html_formatter.ts

Oh that is fine so long as the exception is accessible somewhere along the way. Any pointers as to what needs to be done to pass exceptions up to the formatter constructor or the event collector? I’d be willing to give this a try.