TestNG softAssert in @AfterMethod makes all tests skip

I use a SoftAssert that I throw after my test method, and I use a dataProvider in my test methods running parallel through maven failsafe, it’s running parallel per method. It doesn’t matter if I use parallel=true on my dataprovider or not. Whenever an assert is thrown using my aftermethod, all tests that start in that thread fail immediately because of the dataProvider.

DataProvider:

@DataProvider(name = "environment", parallel = true)
public Object[][] getEnvironments() { return propertiesHelper.getBrowsers() ; }

The dataprovider returns a list of enums, and not null. TestNG also sees no errors in the dataprovider (confirmed through debugging)

Aftermethod:

@AfterMethod(alwaysRun = true)
public void throwSoftAssert() {

    if(SoftAssertThreadLocal.get() == softAssert && softAssert != null) {
            softAssert.assertAll();
    }
}

This is what I see in the log, as can be seen from when my assert fails every test in the thread [pool-2-thread-3] is started and stopped within 1ms, it is then shown as skipped in the test report.

13:23:58.318 [pool-2-thread-3] FATAL nl.adaption.it.selenium_test.util.assertion.SoftAssert - Is seaFreight revenue created? copying D:\Jenkins\workspace\Selenium\workspace\selenium test runner\target\screenshot_Is_seaFreight_revenue_created__1462361038604.png 13:23:58.612 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.dossier.financial.revenue.ITgenerateVesselRevenue - ### STOPPING TEST: whenGeneratingVesselRevenue_RevenuesShouldBeCreated ### 13:23:59.565 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.contract.ITtransportContract - ### STARTING TEST: whenCreatingValidTransportContractWithAgreementAndScenario_ActivitiesShouldBeAutomaticallyCreated ### 13:23:59.566 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.contract.ITtransportContract - ### STOPPING TEST: whenCreatingValidTransportContractWithAgreementAndScenario_ActivitiesShouldBeAutomaticallyCreated ### 13:23:59.569 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.relation.ITuploadRelationLogo - ### STARTING TEST: whenUploadingNewRelationLogo_SpinnerShouldAutomaticallyClose ### 13:23:59.569 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.relation.ITuploadRelationLogo - ### STOPPING TEST: whenUploadingNewRelationLogo_SpinnerShouldAutomaticallyClose ### 13:23:59.571 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.app_currency.ITcreateAppCurrency - ### STARTING TEST: whenCreatingAppCurrencyWithValidData_AndClickingSaveAndClose_AppCurrenciesPageShouldBecomeVisible ### 13:23:59.571 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.app_currency.ITcreateAppCurrency - ### STOPPING TEST: whenCreatingAppCurrencyWithValidData_AndClickingSaveAndClose_AppCurrenciesPageShouldBecomeVisible ### 13:23:59.573 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.app_currency.ITeditAppCurrency - ### STARTING TEST: whenEditingAppCurrencyWithValidData_AndClickingSaveAndClose_AppCurrenciesPageShouldBecomeVisible ### 13:23:59.573 [pool-2-thread-3] INFO nl.adaption.it.selenium_test.tests.app_currency.ITeditAppCurrency - ### STOPPING TEST: whenEditingAppCurrencyWithValidData_AndClickingSaveAndClose_AppCurrenciesPageShouldBecomeVisible ###

[anandbagmar]

@hubertgrzeskowiak - that info helps. I solved the problem the following way:

I made my BaseTest implement IHookable interface. And then implemented the methods as below:

    private static final String SOFT_ASSERT = "softAssert";

    @Override
    public void run(IHookCallBack callBack, ITestResult testResult) {
        SoftAssert softAssert = new SoftAssert();
        testResult.setAttribute(SOFT_ASSERT, softAssert);
        callBack.runTestMethod(testResult);
        softAssert = getSoftAssert(testResult);
        if (!testResult.isSuccess()) {
            Throwable throwable = testResult.getThrowable();
            if (null != throwable) {
                if (null != throwable.getCause()) {
                    throwable = throwable.getCause();
                }
                softAssert.assertNull(throwable, ExceptionUtils.getStackTrace(throwable));
            }
        }
        softAssert.assertAll();
    }

    public static SoftAssert getSoftAssert() {
        return getSoftAssert(Reporter.getCurrentTestResult());
    }

    private static SoftAssert getSoftAssert(ITestResult result) {
        Object object = result.getAttribute(SOFT_ASSERT);
        if (object instanceof SoftAssert) {
            return (SoftAssert) object;
        }
        throw new IllegalStateException("Could not find a soft assertion object");
    }

This solution helps to fail the test with all soft asserts + any hard assert as well.

[hubertgrzeskowiak]

@anandbagmar apologies for not uploading the whole code. My WebDriverTestBase testinstance is having a private list of verification errors. Whenever a verification (a.k.a. soft assert) fails, I add an entry to that list. clearVerificationErrors() is simply clearing the list. assertNoVerificationErrors() is basically assert this.verification_errors.isEmpty() with some fancy printing of the list’s contents.

I can’t give you the original code, because I switched companies since.