Pendings tests

Something that came out of a discussion with Yehuda: During a big refactoring they have hundreds of failing tests. They turned them off using QUnit.skip, but that doesn’t tell them which of the skipped tests starting passing again at some point. To quickly test changes in passing tests, they ended up with this:

QUnit.skip = QUnit.test;

There’s probably a better way to deal with that situation. One idea is to have something like a QUnit.pending method, that has the same signature as QUnit.test(), but will reverse the result. So if all assertions pass, the test fails, if at least one fails, the test passes.

The HTML reporter could use a similar marker as QUnit.skip.

Thoughts?

---

Update 2015-04-15:

After some discussion, we came up with this:

// This test will show up with a "todo" badge, with a similar style as the "skipped" badge
// It will show as passed when at least one assertion fails, or fail when all assertions pass
QUnit.todo( "this isn't implemented, yet" function( assert ) {
  assert.ok( featureTest(), "waiting for feature" );
} );

QUnit.testDone( function( details ) {
  console.log( details.name, details.todo );
  //> "this isn't implemented, yet" true
} );

QUnit.log( function( details ) {
  console.log( details.message, details.todo );
  //> "waiting for feature" true
} );

[wycats]

Incidentally, my first comment here was that if this landed we’d use it. Glimmer is already using it 😃

Thanks for keeping on top of this over a very long development cycle. It’s why I love qunit.

[trentmwillis]

I had considered using “expected”, but that actually doesn’t provide enough contextual information. For example, if you have two expected failures from a todo, but wind up with an unexpected failure it would still be reported as:

{
  "failed": 1,
  "expectedFailures": 2
}

Giving you no way of knowing whether that failure was one of the expected ones or not. Since we’re primarily concerned with the “unexpected” events, reporting those doesn’t have that issue.

I do think consumer code will be simpler if we don’t wrap them in a sub-object.

That’s fine with me, I’m particularly concerned with naming consistency over the exact shape. I’m going to pursue an implementation following like so:

{
  "total": 4,
  "passed": 3,
  "failed": 1,
  "passedUnexpectedly": 1,
  "failedUnexpectedly": 0
}