Flawed tests that pass instead of alerting user of an issue

We have a problem in Chai. The problem is that it’s too easy to write tests that appear valid but are actually broken in such a way that they pass no matter what. Strict TDD/BDD can help mitigate this problem, but it’s not enough, especially when reviewing PRs from developers of varying skill levels.

Before discussing solutions, it’s important to understand the different ways this problem can manifest. Here are three:

describe("flawed tests that always pass instead of alerting user of an issue", () => {

  // Problem A
  it("passes no matter what because of a typo", () => {
    expect(x).to.be.ture;
  });

  // Problem B
  it("passes no matter what because no assertion is actually made", () => {
    expect(x).to.equal;
  });

  // Problem C
  it("passes no matter what because of mishandled asychronicity", () => {
    setTimeout(() => expect(x).to.equal(y), 1000);
  });

});

Commonly proposed solutions include:

1. Replace all property assertions with method assertions (see: dirty-chai).
2. Use ES6 Proxy to detect invalid assertions (see: #721).
3. Add a mechanism to verify that the expected number of assertions were run in each test (see: chai-checkmark).

None of these solutions are mutually exclusive. Here are some pros and cons of each approach:

Replace all property assertions with method assertions

Pros:

• Allows linting tools to be used to automatically identify Problems A and B.
• Decreases chances of Problems A and B from occurring in the first place since developers get used to ending every assertion with a method.
• Makes it easier for reviewers to visually spot Problems A and B for the same reason.
• Easy for Chai to implement.
• Easy for developers to use.
• Works in any version of JavaScript.

Cons:

• Not just a breaking change, but a devastating one, invalidating every single test that uses property assertions. Worse yet, it breaks them silently, causing them to pass no matter what due to the same issue as Problem B. On the plus side, developers who read the patch notes will know to re-enable the linting rule that flags property assertions, but anyone who doesn’t is screwed without them even knowing it.
• Breaks any plugin using property assertions.
• Doesn’t address Problem C.
• Doesn’t actually fix Problems A and B; merely decreases odds of them happening as well as allows the use of a linting tool to detect them.
• Method assertions aren’t as pretty as property assertions.

Use ES6 Proxy to detect invalid assertions

Pros:

• Automatically fixes Problem A.
• Easy for Chai to implement.
• Easy for developers to use.
• Causes few-if-any breaking changes.

Cons:

• Doesn’t address Problems B or C.
• Only works in recent versions of JavaScript; older versions automatically fall back to current Chai functionality.
• Could have performance impact in extreme cases? (unknown)

Add a mechanism to verify that the expected number of assertions were run in each test

Pros:

• Fixes Problems A, B, and C if consumer uses it correctly for each test.
• Causes few-if-any breaking changes.
• Works in any version of JavaScript.

Cons:

• Difficult for Chai to implement because of chaining multiple assertions together, the ability for assertion chains to be branched into separate chains, the tendency for assertions to actually call multiple assertions underneath the hood, and the lack of tight integration between test runner and assertion library.
• Difficult for developers to use because of the above reasons and because they potentially have to add extra code to every test.

Note: A variation of this approach is to merely verify that at least one assertion was run in each test, instead of a specific number, which would be potentially easier for Chai to implement and developers to use, but would mean that it no longer completely fixes any of the problems, only decreases the likelihood of them occurring.

[meeber]

The release of Chai v4 with proxy protection drastically mitigated this problem. Gonna close.

[meeber]

@keithamus Great idea. In fact, I wonder if an attempt to access any property on a non-chainable method assertion should throw an error, even if the property exists. For example expect(x).to.equal.length could throw an error even though the function has a built-in .length property.