Maven BUILD SUCCESSFUL despite exceptions thrown in @Before

Summary

I implemented a custom exception that extends RuntimeException and throw it. I intend the maven build to fail if it is thrown so I do not catch it anywhere. However, mvn clean install still builds successfully despite the custom exception being thrown in @Before.

// Custom exception
@SuppressWarnings("serial")
public class UnsupportedBrowserException extends RuntimeException {
    public UnsupportedBrowserException(String message) {
        super(message);
    }
}


// Method that throws the exception
public final class BrowserFactory {

public static WebDriver getBrowser(String browser) {
    WebDriver driver = null;

    switch (browser.toUpperCase()) {
    ...
    default:
        if (StringUtils.isBlank(browser)) {
            browser = "BLANK";
        }
        throw new UnsupportedBrowserException(ExceptionMessages.UNSUPPORTED_BROWSER + browser);
    }

    return driver;
}


// Call method
import cucumber.api.java.After;
import cucumber.api.java.Before;

public class ScenarioHook {
    private WebDriver driver = null;

    @Before
    public void beforeScenario(Scenario scenario) throws Throwable {
        String browser = PropsLoader.readConfig("browser");
        driver = BrowserFactory.getBrowser(browser);
    }

    @After
    public void afterScenario() {
        driver.quit();
    }
}

Expected Behavior

Maven build is expected to fail if exception is thrown in @Before (or any hook).

Current Behavior

A successful build is returned because tests are being skipped when the exception is thrown within the hook. I also induced a NullPointerException to see if it was my custom exception causing the issue. It appears any exception thrown in @before causes the tests to skip and therefore reports a successful build. Exceptions thrown in non-hook methods promptly fails the build.

[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 5.243 s - in jcucumberng.project.CucumberRunner
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ jcucumberng-framework ---
[INFO] Building jar: D:\dev\automation-testing\java\jCucumberNG-Framework\target\jcucumberng-framework-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-cucumber-reporting:3.17.1:generate (execution) @ jcucumberng-framework ---
[INFO] About to generate Cucumber report.
Jun 23, 2018 9:23:33 PM net.masterthought.cucumber.ReportParser parseJsonFiles
INFO: File 'D:\dev\automation-testing\java\jCucumberNG-Framework\target\cucumber-output\test-report.json' contains 2 features
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ jcucumberng-framework ---
[INFO] Installing D:\dev\automation-testing\java\jCucumberNG-Framework\target\jcucumberng-framework-0.0.1-SNAPSHOT.jar to C:\Users\Kat Rollo\.m2\repository\krollo\jcucumberng\jcucumberng-framework\0.0.1-SNAPSHOT\jcucumberng-framework-0.0.1-SNAPSHOT.jar
[INFO] Installing D:\dev\automation-testing\java\jCucumberNG-Framework\pom.xml to C:\Users\Kat Rollo\.m2\repository\krollo\jcucumberng\jcucumberng-framework\0.0.1-SNAPSHOT\jcucumberng-framework-0.0.1-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.227 s
[INFO] Finished at: 2018-06-23T21:23:34+08:00
[INFO] ------------------------------------------------------------------------

Possible Solution

JUnit @Before fails the build if an exception is thrown.

Context & Motivation

We would usually expect a build failure if an exception is thrown in the hook (since other frameworks do this). It may be worth checking if the same behavior occurs in @after, @beforestep, @afterstep.

Your Environment

• Version used: 3.0.2
• Operating System and version: Win7 x64

[kathyrollo]

@mpkorstanje @mlvandijk

I think I have isolated the issue, I just had another suspicion - the AbstractTestNGCucumberTests in the runner class. I have been using cucumber-testng dependency.

I reverted to cucumber-junit, used the typical @RunWith(Cucumber.class), and removed extends AbstractTestNGCucumberTests. The <testFailureIgnore> config now promptly passes or fails the build with true or false respectively when my custom exception is thrown within @before hook which is the desired behavior.

It may be the AbstractTestNGCucumberTests. Let me know if you arrive at the same conclusion.

P.S. Will be joining the Slack group after my holiday this week. I pushed enhance-33-junit. 😃

[mpkorstanje]

The TestNG reporter/formatter is listening to individual steps and ignoring hooks.

Not sure why it was designed that way.

It should listen test case finished events.

On Mon, Jun 25, 2018, 15:36 Kat Rollo [e-mail] wrote:

@mpkorstanje https://github.com/mpkorstanje @mlvandijk https://github.com/mlvandijk

I think I have isolated the issue, I just had another suspicion - the AbstractTestNGCucumberTests in the runner class. I have been using cucumber-testng dependency.

I reverted to cucumber-junit, used the typical @RunWith(Cucumber.class), and removed extends AbstractTestNGCucumberTests. The <testFailureIgnore> config now promptly passes or fails the build with true or false respectively when my custom exception is thrown within @before https://github.com/before hook which is the desired behavior.

It may be the AbstractTestNGCucumberTests. Let me know if you arrive at the same conclusion.

P.S. Will be joining the Slack group after my holiday this week. 😃

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cucumber/cucumber-jvm/issues/1403#issuecomment-399953873, or mute the thread https://github.com/notifications/unsubscribe-auth/AGoAZ9LinzeKasUdKrnaFkhrkNyKLrF4ks5uAOdEgaJpZM4U0y42 .