gradle ignoring failing tests in a StringSpec

I’m running into an issue with gradle and kotlin-test To verify that tests are executed, I added a broken test. In intellij this runs correctly and it finds test failures. In gradle I get a java.lang.AssertionError: Unexpected start event for [engine:kotlintest]/[spec:StringSpecTest] and the tests are marked as ignored instead of failed. To verify that junit is not the problem, I also added a java test that fail normally. Fixing the tests executes them without the exceptions and they pass.

These tests are ignored unless I fix them. Then they run normally.

package io.inbot.kotlintest

import io.kotlintest.shouldBe
import io.kotlintest.specs.StringSpec

class StringSpecTest : StringSpec({
    "hi wrld" {
        true shouldBe false
    }
    "hi again" {
      // so shouldBe is not the issue
      org.assertj.core.api.Assertions.assertThat(true).isEqualTo(false)
    }
})

These tests fail normally:

public class MyTest {
  @org.junit.jupiter.api.Test
  public void shouldBreak() {
    assert true==false;
  }

  @org.junit.jupiter.api.Test
  public void shouldAlsoBreak() {
    org.assertj.core.api.Assertions.assertThat(true).isEqualTo(false);
  }
}

Full logs & build file below. Let me know if you need more info.

$ gradle -version

------------------------------------------------------------
Gradle 4.7
------------------------------------------------------------

Build time:   2018-04-18 09:09:12 UTC
Revision:     b9a962bf70638332300e7f810689cb2febbd4a6c

Groovy:       2.4.12
Ant:          Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM:          1.8.0_152 (Oracle Corporation 25.152-b16)
OS:           Mac OS X 10.13.4 x86_64

$ rm -rf .gradle/ && gradle build

> Task :test FAILED

Gradle Test Executor 55 STANDARD_OUT
    09:27:30.098 [Test worker] DEBUG io.kotlintest.runner.junit5.KotlinTestEngine - Discovery request [request=org.junit.platform.launcher.core.DefaultDiscoveryRequest@67397d77; uniqueId=[engine:kotlintest]]
    09:27:30.157 [Test worker] DEBUG io.kotlintest.runner.jvm.TestDiscovery - Scan discovered 2 subtypes of Spec...
    09:27:30.399 [Test worker] DEBUG io.kotlintest.runner.jvm.TestDiscovery - ...which has filtered to 1 non abstract classes
    09:27:30.400 [Test worker] DEBUG io.kotlintest.runner.jvm.TestDiscovery - Result is 1 classes after applying extensions

MyTest > shouldBreak() FAILED
    java.lang.AssertionError at MyTest.java:4

MyTest > shouldAlsoBreak() FAILED
    java.lang.AssertionError at MyTest.java:9

Gradle Test Executor 55 STANDARD_OUT
    09:27:30.504 [Test worker] DEBUG io.kotlintest.runner.junit5.KotlinTestEngine - Execution request [org.junit.platform.engine.ExecutionRequest@7ba79419]
    09:27:30.508 [Test worker] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - Engine started; classes=[[class io.inbot.kotlintest.StringSpecTest]]
    09:27:30.509 [Test worker] DEBUG io.kotlintest.runner.jvm.TestEngine - Submitting 1 specs
    09:27:30.510 [Test worker] DEBUG io.kotlintest.runner.jvm.TestEngine - Waiting for spec execution service to terminate
    09:27:30.539 [pool-1-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - prepareSpec [Description(parents=[], name=StringSpecTest)]

io.inbot.kotlintest.StringSpecTest STANDARD_OUT
    09:27:30.554 [pool-2-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - Notifying junit of start event [engine:kotlintest]/[spec:StringSpecTest]/[test:hi wrld]

io.inbot.kotlintest.StringSpecTest > hi wrld STANDARD_OUT
    09:27:30.557 [pool-1-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - completeTestCase Description(parents=[StringSpecTest], name=hi wrld) with result TestResult(status=Failure, error=java.lang.AssertionError: expected: false but was: true, reason=null, metaData={})
    09:27:30.558 [pool-3-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - Notifying junit of start event [engine:kotlintest]/[spec:StringSpecTest]/[test:hi again]

io.inbot.kotlintest.StringSpecTest > hi again STANDARD_OUT
    09:27:30.560 [pool-1-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - completeTestCase Description(parents=[StringSpecTest], name=hi again) with result TestResult(status=Failure, error=java.lang.AssertionError:
    Expecting:
     <true>
    to be equal to:
     <false>
    but was not., reason=null, metaData={})
    09:27:30.560 [pool-1-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - completeSpec [Description(parents=[], name=StringSpecTest)]
    09:27:30.563 [pool-1-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - Notifying junit of completion event [engine:kotlintest]/[spec:StringSpecTest]/[test:hi again]=TestResult(status=Failure, error=java.lang.AssertionError:
    Expecting:
     <true>
    to be equal to:
     <false>
    but was not., reason=null, metaData={})
    09:27:30.565 [pool-1-thread-1] ERROR io.kotlintest.runner.junit5.JUnitTestRunnerListener - Error in JUnit Platform listener
    java.lang.AssertionError: Unexpected start event for [engine:kotlintest]/[spec:StringSpecTest]/[test:hi again]
        at org.gradle.api.internal.tasks.testing.junit.GenericJUnitTestEventAdapter.testStarted(GenericJUnitTestEventAdapter.java:52)
        at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener.executionFinishedAfterTestClassStart(JUnitPlatformTestExecutionListener.java:107)
        at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener.executionFinished(JUnitPlatformTestExecutionListener.java:91)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry$CompositeTestExecutionListener.lambda$executionFinished$3(TestExecutionListenerRegistry.java:77)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry.notifyTestExecutionListeners(TestExecutionListenerRegistry.java:51)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry.access$100(TestExecutionListenerRegistry.java:27)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry$CompositeTestExecutionListener.executionFinished(TestExecutionListenerRegistry.java:77)
        at org.junit.platform.launcher.core.ExecutionListenerAdapter.executionFinished(ExecutionListenerAdapter.java:56)
        at io.kotlintest.runner.junit5.SynchronizedEngineExecutionListener.executionFinished(SynchronizedEngineExecutionListener.kt:12)
        at io.kotlintest.runner.junit5.JUnitTestRunnerListener.completeSpec(JUnitTestRunnerListener.kt:127)
        at io.kotlintest.runner.jvm.TestEngine.executeSpec(TestEngine.kt:112)
        at io.kotlintest.runner.jvm.TestEngine.access$executeSpec(TestEngine.kt:14)
        at io.kotlintest.runner.jvm.TestEngine$submitSpec$1.run(TestEngine.kt:89)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    09:27:30.566 [pool-1-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - Notifying junit of completion event [engine:kotlintest]/[spec:StringSpecTest]/[test:hi wrld]=TestResult(status=Failure, error=java.lang.AssertionError: expected: false but was: true, reason=null, metaData={})
    09:27:30.567 [pool-1-thread-1] ERROR io.kotlintest.runner.junit5.JUnitTestRunnerListener - Error in JUnit Platform listener
    java.lang.AssertionError: Unexpected start event for [engine:kotlintest]/[spec:StringSpecTest]/[test:hi wrld]
        at org.gradle.api.internal.tasks.testing.junit.GenericJUnitTestEventAdapter.testStarted(GenericJUnitTestEventAdapter.java:52)
        at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener.executionFinishedAfterTestClassStart(JUnitPlatformTestExecutionListener.java:107)
        at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestExecutionListener.executionFinished(JUnitPlatformTestExecutionListener.java:91)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry$CompositeTestExecutionListener.lambda$executionFinished$3(TestExecutionListenerRegistry.java:77)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry.notifyTestExecutionListeners(TestExecutionListenerRegistry.java:51)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry.access$100(TestExecutionListenerRegistry.java:27)
        at org.junit.platform.launcher.core.TestExecutionListenerRegistry$CompositeTestExecutionListener.executionFinished(TestExecutionListenerRegistry.java:77)
        at org.junit.platform.launcher.core.ExecutionListenerAdapter.executionFinished(ExecutionListenerAdapter.java:56)
        at io.kotlintest.runner.junit5.SynchronizedEngineExecutionListener.executionFinished(SynchronizedEngineExecutionListener.kt:12)
        at io.kotlintest.runner.junit5.JUnitTestRunnerListener.completeSpec(JUnitTestRunnerListener.kt:127)
        at io.kotlintest.runner.jvm.TestEngine.executeSpec(TestEngine.kt:112)
        at io.kotlintest.runner.jvm.TestEngine.access$executeSpec(TestEngine.kt:14)
        at io.kotlintest.runner.jvm.TestEngine$submitSpec$1.run(TestEngine.kt:89)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    09:27:30.568 [pool-1-thread-1] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - Notifying junit that spec finished [engine:kotlintest]/[spec:StringSpecTest]=TestExecutionResult [status = SUCCESSFUL, throwable = null]

Gradle Test Executor 55 STANDARD_OUT
    09:27:30.570 [Test worker] DEBUG io.kotlintest.runner.junit5.JUnitTestRunnerListener - Engine finished; throwable=[null]

4 tests completed, 2 failed, 2 skipped

I looked at the code where the exception is thrown:

    public void testStarted(T identifier, TestDescriptorInternal descriptor) {
        synchronized (lock) {
            TestDescriptorInternal oldTest = executing.put(identifier, descriptor);
            assert oldTest == null : String.format("Unexpected start event for %s", identifier);
        }
        resultProcessor.started(descriptor, startEvent());
}

It looks like gradle is somehow trying finding an old test run.

here’s my build file. I’ve tried to strip as much as possible to produce the issues. Let me know if you need more. The gradle version is

buildscript {
    ext.kotlinVersion = '1.2.41'
    ext.junitVersion = '5.2.0'

    repositories {
        mavenCentral()
        jcenter()
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlinVersion"
    }
}

apply plugin: 'java'
apply plugin: 'kotlin'

tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile).all {
    kotlinOptions {
        // for whatever reason defaults to 1.6. But since we don't care about android ...
        jvmTarget = "1.8"
    }
}

sourceSets {
    main.java.srcDirs += 'src/main/kotlin'
    test.java.srcDirs += 'src/test/kotlin'
}

repositories {
    mavenCentral()
}

test {
    // Enable JUnit 5 (Gradle 4.6+).
    useJUnitPlatform()
    testLogging {
        // Make sure output from
        // standard out or error is shown
        // in Gradle output.
        showStandardStreams = true
    }
}

dependencies {
    // needed to make test logging work, removing this and test logging does not solve the problem
    compile "ch.qos.logback:logback-classic:1.2.3"
    // not the problem either but use this to verify shouldBe is not the problem; assertJ fails in the same way but only in kotlin-test
    testCompile 'org.jboss.forge.addon:assertj:3.6.2'

    testCompile "org.junit.jupiter:junit-jupiter-api:${junitVersion}"
    testCompile "org.junit.jupiter:junit-jupiter-engine:${junitVersion}"
    testCompile 'io.kotlintest:kotlintest-runner-junit5:3.1.1'
}

[sksamuel]

3.1.5 is available now.

[sksamuel]

@jillesvangurp @sschuberth I’ve added workaround that seems to work fine in both gradle 4.6 and 4.7 and keep intellij functionality the same. So I’m happy with that. I’ve released 3.1.5 with this fix applied. I’d love it if you could test it in case my unit tests and manual testing didn’t cover everything. Please re-open if the problem still occurs for you.