TestNG - configfailurepolicy=“continue” is not working for retried test

TestNG Version

7.0.0

Expected behavior

As configfailurepolicy is configured as “continue”, the failure in @AfterMethod shouldn’t cause retried test to be skipped.

Actual behavior

Though configfailurepolicy value is set as “continue”, the failure in @AfterMethod is causing the the retried test to be skipped.

Is the issue reproductible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

testng.xml - At suite level, configfailurepolicy value is set as “continue”

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="classes" thread-count="1"
	configfailurepolicy="continue">
	<test name="Front-End">
		<classes>
			<class
				name="com.tests.ExcepionAfterMethodTest">
			</class>
		</classes>
	</test>
</suite>

Test: Throwing an exception in configuration (i.e., @AfterMethod) method to cause failure. And using RetryAnalyzer to retry the failed test.

import org.testng.Assert;
import org.testng.annotations.Test;

public class ExcepionAfterMethodTest{

    @BeforeMethod(alwaysRun=true)
    public void beforeMethod() {
        System.out.println("Before Method");
    }
    
    @Test (alwaysRun=true, retryAnalyzer=RetryAnalyzer.class)
    public void method() {
        System.out.println("Method");
        Assert.assertTrue(false);
    }

    @Test (alwaysRun=true)
    public void method1() {
        System.out.println("Method1");
    }

    @AfterMethod(alwaysRun=true)
    public void afterMethod() throws Exception {
        System.out.println("After Method");
        throw new Exception();
    }
}

RetryAnalyzer: Will retry once, if a failure occurs.

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {

    private int counter = 0;
    private int retryLimit = 1;

    @Override
    public boolean retry(ITestResult result) {
        if (counter < retryLimit) {
            counter++;
            return true;
        }
        return false;
    }
}

Result: Because of failure in @AfterMethod, the test is retried, but the @Test is skipped. Hence, ‘Method’ is not printed in the result at the 5th line.

Before Method
Method
After Method
Before Method
After Method
Method1
After Method

Expected Result: As configfailurepolicy is configured as “continue”, the retried test shouldn’t be skipped. So, expecting ‘Method’ should be printed in the result at the 5th line as below.

Before Method
Method
After Method
Before Method
Method
After Method
Method1
After Method

[krmahadevan]

@Proryanator - You can do one of the following:

1. If you are using the maven-surefire plugin, then you can pass this as a property

<configuration>
	<properties>
		<property>
			<name>configfailurepolicy</name>
			<value>continue</value>
		</property>
	</properties>
</configuration>

1. The other option is to basically build an implementation of org.testng.IAlterSuiteListener and then wire in via service loading mechanism as detailed here

[Proryanator]

Is there a way to set the configfailurepolicy without having a suite file from our test code?

I do see that you can pass in a parameter of ‘-DconfigFailurePolicy’ and modify it that way. I don’t think that’s working though.

We removed the suite file long back for our project since it caused us to have to add in a test every time we made a new one, or remember to modify it if it gets re-named, etc. We just have maven run every @Test annotated method which works super well for us.

Would love to know if there’s some way to modify/adjust this property from a method listener or something.