How to check the test case corresponding to the invocation-numbers value from testng-failed.xml?

TestNG Version

7.4.0

Expected behavior

The test that failed in the first run should be the one that is retried.

Actual behavior

Some other test is run in the retry using the testng-failed.xml file.

Is the issue reproducible on runner?

• Shell
• Maven
• [✓] Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

I’m having a bunch of tests that I run in parallel with a dataproviderthreadcount of 10. I see that the test that fails in the first run is not the same as that run in a retry using the testng-failed.xml file that looks like below. I believe the invocation-numbers property value is used for retrying the failed tests. Since I observe the issue with a 100% replication rate, I’d first like to understand how to identify the test case corresponding to this value.

<suite thread-count="10" parallel="classes" name="Failed suite [Gradle suite]" verbose="0">
<test thread-count="10" parallel="classes" name="Gradle test(failed)" verbose="0">
<groups>
<run>
<include name="GROUP_1"/>
<include name="GROUP_2"/>
</run>
</groups>
<classes>
<class name="package.TestSample">
<methods>
<include name="beforeSuiteHook"/>
<include name="verifySomething" invocation-numbers="9"/>
<include name="beforeMethodHook"/>
<include name="afterSuiteHook"/>
</methods>
</class>
<!--  package.TestSample  -->
</classes>
</test>
<!--  Gradle test(failed)  -->
</suite>
<!--  Failed suite [Gradle suite]  -->

[krmahadevan]

do you mean the point 3 in that comment isn’t taken care of in your code sample?

Yes.

I’m wondering if there’s a simpler way to log all the test records corresponding to all the invocation-numbers. Since I see that the test failed is not the one retried, I’d like to check if the testng-failed.xml has correct invocation-numbers in the first place.

There does not exist anything today. Which was why I shared a sample that shows you how to do it.

[krmahadevan]

Does it also mean the retry attempt with such a custom generated testng-failed.xml file as an input will not work since it does not have invocation-numbers (asking this since the sample generated testng suite xml does not have invocation-numbers)?

That was a file that I created manually for demonstration purposes. If you are generating the TestNG suite xml file wherein you would like to build the intelligence of a retry of the failed invocation counts, then you would need to ensure that you include these values as well. What I shared is not meant to be used as a “use as is solution” but something that you would need to work on and tweak. So please try it out and then raise any specific issues that you may have encountered.

If this has answered your question, please feel free to close this issue.