System Exit Extension: Test fails though it should pass

Kotest 4.4.2

I encountered strange behaviour when using the System Exit Extension:

Minimum working example:

import io.kotest.assertions.throwables.shouldThrow
import io.kotest.core.spec.style.StringSpec
import io.kotest.extensions.system.*
import io.kotest.matchers.shouldBe
import picocli.CommandLine
import picocli.CommandLine.Command
import java.util.concurrent.Callable
import kotlin.system.exitProcess

class SystemExitTest : StringSpec() {

    override fun listeners() = listOf(SpecSystemExitListener)

    init {
        "Test passes as expected" {
            shouldThrow<SystemExitException> {
                exitProcess(CommandLine(PicocliApp()).execute())
            }.exitCode shouldBe 42
        }

        "Test fails though it should pass" {
            shouldThrow<SystemExitException> {
                CommandLine(PicocliApp2()).execute()
            }.exitCode shouldBe 42
        }
    }
}

@Command
class PicocliApp : Callable<Int> {
    override fun call(): Int {
        return 42
    }
}

@Command
class PicocliApp2 : Callable<Int> {
    override fun call(): Int {
        exitProcess(42)
    }
}

I’m testing a command line application based on the https://picocli.info framework.

First test: PicocliApp app is called, returns an exit code, top level process exits with returned code. Everything is fine here.

Second test: PicocliApp app is called, app exits inside the overriden call function. Test fails though it should pass.

I tested the both same tests using Stefan Birkner’s System rules. Result: both tests pass (as expected).

I admit that it is a bit unusual to exit from the call function of a command. Certainly the better way of coding is following the convention to return an exit code and call System.exit from the main method with the return value of the execute method. Nonetheless, test 2 contains valid code which should be handled correctly from the test framework.

Thanks for investigating!

[remkop]

An alternative would be to throw a Throwable from the security manager instead of Exception, because my guess is that libraries and applications are less likely to catch all Throwables (and cause the same issue with the test). However, “less likely” is not zero. 😃

The approach suggested by @sksamuel to store the last return code so it can be inspected is more robust.

Thanks all for raising this and the analysis! And nice work with the kotest library! 😃

[deining]

What we can do is improve the listener to record every caught System.exit

Doesn’t that make sense?

I see it from a test user perspective: I called System.exit and want to test for this, but the test fails. Like said, other test systems perform different (=better) when confronted with my second test case.

You may also have a look at the perspective from the author of picocli.