RetryAnalyzer doesn't retry @Before* methods that fail

TestNG Version

7.0.0

Expected behavior

From what I was told in this post, it was designed that way: https://groups.google.com/forum/#!topic/testng-users/64YfIs1WjAg

However, I would suggest adding a feature to allow people to also retry @Before* methods that fail.

You may be asking why do we need this feature? The answer is because for very complex software ecosystems (mine is composed of about 27 different VMs with different micro-services on them) environmental flakiness is inevitable. Sometimes there may be timeouts or messages could get delayed in some parts of the system that could cause some of the test setup methods to fail; and simply retrying them could allow them to be successful, thereby allowing the rest of the tests to run normally. Essentially, it’s the same reason why the RetryAnalyzer retries @Test methods that fail. Flaky tests are inevitable in large systems.

Actual behavior

It looks like if the @Test method gets run and fails, it will retry the @BeforeMethod method before retrying the @Test method, but if the @BeforeMethod fails before the @Test runs, it won’t get retried. Also, if the @BeforeSuite, @BeforeClass or @BeforeGroups methods fail they don’t get retried either.

Is the issue reproductible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

public class Utility
{
    public static String getClassName() {
        StackTraceElement stElement = Thread.currentThread().getStackTrace()[2];
        return stElement.getClassName();
    }

    public static String getFunctionName() {
        StackTraceElement stElement = Thread.currentThread().getStackTrace()[2];
        return stElement.getMethodName();
    }

    public static String getClassAndFunctionName() {
        StackTraceElement stElement = Thread.currentThread().getStackTrace()[2];
        return stElement.getClassName() + "." + stElement.getMethodName();
    }
}

You can set different fail* boolean variables to true to see how it behaves for each type of failure.

@Listeners({ MyListener.class })
public class TestMyListener
{
    private static final Logger LOG = Logger.getLogger(TestMyListener.class);

    private static Map<String, Integer> methodCalls = new HashMap<>();

    private static boolean failBeforeSuite = false;
    private static boolean failBeforeTest = false;
    private static boolean failBeforeClass = false;
    private static boolean failBeforeGroups = false;
    private static boolean failBeforeMethod = true;
    private static boolean failTest1 = false;
    private static boolean failAfterMethod = false;
    private static boolean failAfterGroups = false;
    private static boolean failAfterClass = false;
    private static boolean failAfterTest = false;
    private static boolean failAfterSuite = false;

    private void runFunction(String name, boolean shouldFail) {
        final String msg = "Called: " + name + " at timestamp: " + ZonedDateTime.now().toInstant().toEpochMilli();
        LOG.info(msg);

        if (methodCalls.containsKey(name)) {
            methodCalls.put(name, methodCalls.get(name) + 1);
        } else {
            methodCalls.put(name, 1);
        }

        if (shouldFail && (methodCalls.get(name).intValue() == 1)) {
            final String err = "Boom in: " + name;
            LOG.error(err);
            Assert.fail(err);
        }
    }

    @BeforeSuite(alwaysRun = true)
    public void beforeSuite() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failBeforeSuite);
    }

    @BeforeTest(alwaysRun = true)
    public void beforeTest() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failBeforeTest);
    }

    @BeforeClass(alwaysRun = true)
    public void beforeClass() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failBeforeClass);
    }

    @BeforeGroups("group1")
    public void beforeGroups() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failBeforeGroups);
    }

    @BeforeMethod(alwaysRun = true)
    public void beforeMethod() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failBeforeMethod);
    }

    /////////////////////////////////////////////////////////////////

    @Test(groups = "group1", retryAnalyzer = RetryAnalyzer.class, alwaysRun = true)
    public void test1() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failTest1);
    }

    /////////////////////////////////////////////////////////////////

    @AfterMethod(alwaysRun = true)
    public void afterMethod() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failAfterMethod);
    }

    @AfterGroups("group1")
    public void afterGroups() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failAfterGroups);
    }

    @AfterClass(alwaysRun = true)
    public void afterClass() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failAfterClass);
    }

    @AfterTest(alwaysRun = true)
    public void afterTest() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failAfterTest);
    }

    @AfterSuite(alwaysRun = true)
    public void afterSuite() {
        LOG.trace("*** Called: " + Utility.getClassAndFunctionName());
        runFunction(Utility.getFunctionName(), failAfterSuite);

        methodCalls.forEach((String name, Integer calls) -> LOG.info("### Called " + name + "() " + calls + " times."));
    }
}

[krmahadevan]

@cpjust - 7.1.0 was our last release. We are working on getting 7.2.0 release done in a week or two.

If you are talking about snapshots getting published, we used to have that. For some reason it seems to have stopped. I am trying to find out what happened.

[krmahadevan]

@cpjust - Here’s a lesser invasive, Do it yourself option by leveraging org.testng.IConfigurable

public class TestClassSample implements IConfigurable {

  private int counter = 1;

  @Override
  public void run(IConfigureCallBack callBack, ITestResult testResult) {
    callBack.runConfigurationMethod(testResult);
    if (testResult.getThrowable() != null) {
      for (int i = 0; i <= 3; i++) {
        callBack.runConfigurationMethod(testResult);
        if (testResult.getThrowable() == null) {
          break;
        }
      }
    }
  }
}

The above alternative along with a PR ( #2262 ) that I have raised should collectively let you retry configurations.