Per-leaf test isolation

I’ve been thinking about possible improvements to kotlintest’s runner structure for the next major version of the library.

Currently, kotlintest makes it easy to hierarchically structure test cases, but setting up test state requires listeners, which are more cumbersome. Since the entire spec is executed in a single pass, you can’t set up test state inside a spec without it being shared between each test.

Listeners work, and are reusable. But in my experience, sharing setup and teardown code between test classes is uncommon outside of a few junit rules like creating temporary files. Tests with the same setup are usually all in the same class.

My favorite hierarchical test runner is Catch2 for C++. Tests in Catch2 are structured similarly to kotlintest, with top-level test cases that contain nested sections that form a tree of tests. The big difference from kotlintest is in how the control flow works. Quoting the Catch2 docs:

Sections can be nested to an arbitrary depth (limited only by your stack size). Each leaf section (i.e. a section that contains no nested sections) will be executed exactly once, on a separate path of execution from any other leaf section (so no leaf section can interfere with another). A failure in a parent section will prevent nested sections from running - but then that’s the idea.

Example

Let’s say we want to do an integration test of a database model where each test is run inside a transaction that is rolled back after the test ends:

class UserTableTest: KotlintestSpec({
    val db = Database.connectToDatabase()
    db.transaction { transaction ->
        section("an existing user") {
            val user = db.insertNewUser()
            section("a new user is added") {
                db.insertNewUser()
                db.fetchUsers().size shouldBe 2
            }
            section("the user is deleted") {
                db.delete(user)
                db.fetchUsers().size shouldBe 0
            }
        }
        
        section("two related users") {
            // etc ...
        }

        transaction.rollback()
    }
    db.close()
})

All of the setup and teardown is declared naturally with zero boilerplate. Since the entire spec is run once for each leaf section, we don’t have to worry about test cases interfering with each other. This, in my opinion, a big improvement over JUnit rules, test listeners, etc.

Implementation

So how would we implement it? Catch2 takes advantage of macros and templates to set up its magic. We don’t have those tools in Kotlin, so we would probably have to implement a runtime solution.

Here’s an algorithm that we could use: https://gist.github.com/ajalt/df82eb9a32383c720f22081f449781f7

This algorithm allows sections to be nested arbitrarily. An advantage of this is that all the different test styles (like given/when/then, feature/scenario/on/it, String.() -> Unit etc.), could be implemented as aliases for section, obviating the need for the many current spec classes.

Fixtures

With this design, running code before and after each test is easy, as shown above. But we still need to consider how to run code before and after the entire spec. In the example above, if the database connection is expensive to set up, it would be nice to only connect once for the entire spec, and reuse the connection for each test. Here are two possible APIs for this:

Spec({
    val db = runOnce { Database.connectToDatabase() }
    section("...") {
        // ...
    }
    teardown {
        db.close()
    }
})

Spec({
    val db = runOnce(
        setup = { Database.connectToDatabase() }
        teardown = { db -> db.close() }
    ) 
    section("...") {
        // ...
    }
})

I feel that that second option is less error-prone. Both options could be implemented be caching the object and teardown block on the spec object.

Drawbacks

The runtime implementation does bring some challenges.

Excluding individual sections or tests is easy, but specifying a single test (or group of tagged tests) to run is more challenging. Since we don’t know whether any given section contains the specified test until we run it, we wouldn’t be able to support adding “f:” to the start of test names. Additionally, if you add a “!” to the name of a leaf section, we don’t know not to run the outer sections that contain the excluded test. You would have to specify the entire path to the section that you want to exclude/focus on before the spec is run. This is easy to do for IDE plugins, but is probably cumbersome from the command line.

Let me know what you think. Is this viable? It seem like it’s the direction that Kotlintest and Spek are going, but without the gotchas of Spek or the boilerplate of test listeners. It does make the common case of running a single test more difficult, however.

[sksamuel]

I’ve added an InstancePerLeafSpecRunner which executes each leaf node in turn, in it’s own instance. As part of implementing this, and looking at your gist, I realised what you want is different to what I thought you wanted.

So in previous versions of KotlinTest we’ve had two modes:

• Run each test as it registered, in order, all in a single instance of the spec class (1)
• Run each test as it registered, in order, with the path to the newly registered test + the test running in a clean instance of the spec class. (2)

(I’m not sure if there’s a term for this, but I will call a path from the root to an edge (leaf) a terminal path.)

And now I’ve implemented

• Run each terminal path, in isolation, with each terminal path executing in a clean instance of the spec. (3)

I think what you wanted was

• Run each terminal path, in isolation, in the same instance of the spec. (4)

Which of course I can implement as well - would be simpler than the one I’ve just done. I also think mode (2) is probably pretty useless. If you want a fresh instance of the test class (which is what junit does and hence why it’s popular), then you probably want an entire path (either terminal or not) to complete before a new path is started. And mode 4 is nice too (your requirement).

So now, the newly minted isolation mode has three values:

enum class SpecIsolationMode {
  SharedInstance,
  InstancePerNode,
  InstancePerLeaf
}

But it needs to cover the new case. And perhaps all need to be renamed.

[ajalt]

Awesome work! I agree with you that mode (2) probably doesn’t have any use-cases, and could be deprecated/removed. You’re correct that I had originally envisioned case (4), but looking at it now, I think that (3) and (4) would both work equally well, so I don’t think there’s any reason to implement both.