question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

TestNG swallows exceptions silently if @ClassRule is used

See original GitHub issue

Testcase:

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.mycompany</groupId>
    <artifactId>JUnitTestcase</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.9.8</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.19.1</version>
                <configuration>
                    <trimStackTrace>false</trimStackTrace>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

MyTest.java

import org.junit.ClassRule;
import org.junit.Test;
import org.junit.rules.ExternalResource;

public class MyTest {

    @ClassRule
    public static ExternalResource resource = new ExternalResource() {
        @Override
        protected void before() throws Throwable {
            throw new IllegalArgumentException("before");
        }

        @Override
        protected void after() {
            throw new IllegalArgumentException("after");
        }
    };

    @Test
    public void myTest() {
        System.out.println("yay!");
    }
}

If I disable TestNG in the pom file I get:

Running MyTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.064 sec <<< FAILURE! - in MyTest
MyTest  Time elapsed: 0.063 sec  <<< ERROR!
java.lang.IllegalArgumentException: before
    at MyTest$1.before(MyTest.java:12)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)

If I enable TestNG I get:

Running MyTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.69 sec - in MyTest

I am expecting to get similar output whether the TestNG plugin is enabled or not.

The popular Dropwizard framework relies on @ClassRule so any minor configuration problem leads to unit tests silently failing.

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:5

github_iconTop GitHub Comments

2reactions
cowwoccommented, May 18, 2016

For anyone running into this in the future, here is how I worked around the problem (hat tip @juherr):

<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<dependencies>
    <!-- WORKAROUND: https://github.com/cbeust/testng/issues/1045#issuecomment-220086979 -->
    <dependency>
    <groupId>org.apache.maven.surefire</groupId>
    <artifactId>surefire-junit4</artifactId>
    <version>2.19.1</version>
    </dependency>
</dependencies>
0reactions
juherrcommented, May 30, 2016

The issue comes from https://github.com/cbeust/testng/blob/285cf3d9825eb4ed6f05a35c1626c989e5aece96/src/main/java/org/testng/junit/JUnit4TestRunner.java#L137

In case of an exception into a @ClassRule (but into a @BeforeClass too), tr is null because the description is not related to a test. TestNG has a specific listener for configuration methods but JUnit has not and I have not enough context information in the object. BTW, I need to find a way to manage it.

From TestNG point-of-view, the output is:

Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 1

From JUnit point-of-view, the output is:

Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 2, Skips: 0

@cbeust If you have an idea, I’m open 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

TestNG swallows exception - java - Stack Overflow
In my method below when I get a IOException when I run it from TestNG (this is expected) but all I see is...
Read more >
JUnit 5 User Guide
With the exception of @Test , these create a container in the test ... will be used; any additional declarations will be silently...
Read more >
Swallowed Exceptions: The Silent Killer of Java Applications
In this post, we're looking at understanding the negative impact of swallowed exceptions and learning how to fix them.
Read more >
The Great Transformation - INCT/PPED
The great transformation: the political and economic origins of our time / Karl ... to politics, religion, and social relations.10 Polanyi's use of...
Read more >
News — JRuby.org
Empty rescue block returns exception object instead of nil. ... This improves integration with Java frameworks that use reflection to access object state....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found