Expect test to fail (xfail style)

🚀 Feature Proposal

The ability to mark a test as expected to fail probably with the following syntax:

describe('Test isCool function works', {
  test('jeff is the best', () => {
    expect(isCool("jeff")).toBe(true);
  });

  test.xfail("for some reason I'm not coming out as cool...", () => {
    expect(isCool("will")).toBe(true);
  });
});

And suggested output:

Test Suites: 1 failed, 5 passed 6 total
Tests:       1 failed, 16 passed, 2 expected failures, 2 unexpected passes, 21 total
Snapshots:   0 total
Time:        2.627s, estimated 4s
Ran all test suites.

Motivation

Many testing frameworks allow you to specify a test as a expected failure. This can be very useful for long-term TDD and for testing known-bugs are still bugs without implying that it’s intended behaviour.

Example

In the above output, an engineer should be surprised (and pleased) by the 2 unexpected passes and change those tests to test from test.xfail if their commit caused the fixes. Otherwise they can leave as is for an engineer who understands those tests to mark as resolved.

Pitch

I feel this is a core feature as it’s a fundamentally new result status for tests.

See some great comments from an older issue #8317

[snoozbuster]

+1 on this. Sometimes we have frontend functionality which needs to be hard-disabled due to a backend/API bug. Often times this is code which is tested and works, but the data isn’t reliable enough to show it to customers. After stubbing the code we can use it.skip we can disable these tests, but there is nothing which forces them to be re-enabled after the upstream services are working and the code is re-enabled (someone unaware of the test may leave the it.skip in - after all, “the tests are passing”). An xfail like interface would start failing after the feature was re-enabled in this case, ensuring that the test is also turned on in conjunction.

[dgkimpton]

I just came here to make a similar request, I was going to call it test.quarantine but the functionality is the same. I have a test that is currently broken and I accept that but would like to know if something happens to make it pass. So ideally this would make a failing test pass until it would actually pass, at which point it should fail.

updated: I previously added some workarounds, none of which were very good. I’ve since come up with a “better” one (although it’s pretty unpleasant code).

test.quarantine = function (description, func) {
  describe("in quarantine", () => {
    try {
      func()
      test(description, () => {
        const e = new Error("[" + description + "] was expected to fail, but it passed")
        e.name = 'Quarantine Escape'
        throw e
      })
    } catch (e) {
      test.todo('fix ' + description)
    }
  })
}

used as

  test.quarantine('testA', () => {
    expect(false).toBe(true)
  })
  test.quarantine('testB', () => {
    expect(false).toBe(false)
  })

which shows

in quarantine                                                                                                                                                                                                                                                                                             
      × testB (1 ms)                                                                                                                                                                                                                                                                                          
      ✎ todo fix testA      

I’ll leave this here in case it’s useful to anyone.