TestNG swallows exceptions silently if @ClassRule is used

Testcase:

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mycompany</groupId>
    <artifactId>JUnitTestcase</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.9.8</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.19.1</version>
                <configuration>
                    <trimStackTrace>false</trimStackTrace>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

MyTest.java

import org.junit.ClassRule;
import org.junit.Test;
import org.junit.rules.ExternalResource;

public class MyTest {

    @ClassRule
    public static ExternalResource resource = new ExternalResource() {
        @Override
        protected void before() throws Throwable {
            throw new IllegalArgumentException("before");
        }

        @Override
        protected void after() {
            throw new IllegalArgumentException("after");
        }
    };

    @Test
    public void myTest() {
        System.out.println("yay!");
    }
}

If I disable TestNG in the pom file I get:

Running MyTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.064 sec <<< FAILURE! - in MyTest
MyTest  Time elapsed: 0.063 sec  <<< ERROR!
java.lang.IllegalArgumentException: before
    at MyTest$1.before(MyTest.java:12)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)

If I enable TestNG I get:

Running MyTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.69 sec - in MyTest

I am expecting to get similar output whether the TestNG plugin is enabled or not.

The popular Dropwizard framework relies on @ClassRule so any minor configuration problem leads to unit tests silently failing.

[cowwoc]

For anyone running into this in the future, here is how I worked around the problem (hat tip @juherr):

<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<dependencies>
    <!-- WORKAROUND: https://github.com/cbeust/testng/issues/1045#issuecomment-220086979 -->
    <dependency>
    <groupId>org.apache.maven.surefire</groupId>
    <artifactId>surefire-junit4</artifactId>
    <version>2.19.1</version>
    </dependency>
</dependencies>

[juherr]

The issue comes from https://github.com/cbeust/testng/blob/285cf3d9825eb4ed6f05a35c1626c989e5aece96/src/main/java/org/testng/junit/JUnit4TestRunner.java#L137

In case of an exception into a @ClassRule (but into a @BeforeClass too), tr is null because the description is not related to a test. TestNG has a specific listener for configuration methods but JUnit has not and I have not enough context information in the object. BTW, I need to find a way to manage it.

From TestNG point-of-view, the output is:

Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 1

From JUnit point-of-view, the output is:

Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 2, Skips: 0

@cbeust If you have an idea, I’m open 😃