Expect test to fail

🚀 Feature Proposal

Hi folks, I would like there to be a way to write a test that is expected to fail. Essentially, this would flip the outcome of the test from success to failure and vice versa.

Motivation

This came up when I was writing some code to auto-fail tests that call console.error. I want to verify that tests that call console.error fail, but there is no way to do so–the test is supposed to fail, but failed tests would in turn break my build. Note that unfortunately there is no way to write this using an expect and condition–I am programmatically failing the tests using expect.assertions(Infinity), so there’s nothing I could mock out.

Example

describe('Ensure tests fail when they're supposed to', {
  itShouldFail('because 1 is not 0', () => {
    expect(1).toBe(0);
  });
});

Pitch

One of the scariest things as an engineer is a test that passes when it should fail. This is unfortunately pretty common when dealing with asynchronous Javascript. As such it’s occasionally a good idea to ensure that a certain kind of behavior always generates a test failure. While the above example may seem trivial, it’s actually useful to ensure that tests that are supposed to fail actually fail.

This would also be useful for testing the internal behavior of Jest.

[alistairnewpark]

XFAIL is often used to mark a bug that is unfixed. The test is in place, but the fix is not done yet. I imagine something like:

test.xfail(…);

Similar to the test.todo(…); place holder and test.skip(…);.

test.xfail(…); would show as a separate row in the test results, although it is treated as a pass for the purpose of continuing the test run. When an XFAIL test passes then it is marked as XPASS in the test results, but a fail for the test run. This highlights the test is now working and the test should be changed from test.xfail(…); to test(…);.

For TAP output an XFAIL is shown as: not ok 10 # TODO See issue #123456789 and an XPASS as: ok 10 # TODO See issue #123456789 For reference see https://testanything.org/tap-version-13-specification.html#todo-tests

This would be a good feature to have in Jest.

This issue gives a negative test as an example and I agree that should not be a reason for using XFAIL.

[vith]

ava also has such a feature:

// See: github.com/user/repo/issues/1234
test.failing('demonstrate some bug', t => {
	t.fail(); // Test will count as passed
});

quite useful. i must have checked the jest docs for it on 4 or 5 occasions now 😦

the ability to parse the test results with some external tool is not really a replacement for the use case of contributing a known-failing test as a way of reporting a bug.

https://github.com/avajs/ava/blob/master/docs/01-writing-tests.md#failing-tests https://github.com/avajs/ava