IRetryAnalyzer is broken

I tend to use a lot of TestNG’s more advanced features for integration testing, and I’d really like to be able to use IRetryAnalyzer with inherently non-deterministic tests that can fail spuriously. Unfortunately that feature appears to be fundamentally broken because of how it was integrated with TestNG’s various callback APIs:

• The Reporter(s) will be notified of each FAILED invocation attempt, even if the test method ultimately succeeds
• Similarly, the ExitCodeListener in org.testng.TestNG calls m_mainRunner.setStatus(FAILED) as soon as a test invocation fails (the test method’s ultimate success does not matter), so TestNG will exit with a nonzero exit code (resulting in, for instance, the overall Ant or Maven task failing), even if you use one of the hacks intended to fix the test report

I’m only aware of two ways around these fundamental deficiencies: reflection hacks to mutate the internal state of the TestNG instance, or writing a custom test runner to replace it.

I’m willing to do the design and coding work to fix this issue, but first I want to make sure everyone agrees with my statement of the problem.

[nmiroshnichenko]

And I think “failures-to-be-retried” should be reported (with some “non-final-failure-status”), user can ignore such status, but other users wish to track such non-final-failures. For example, we log such failures as ordinary failures (with stack traces and screenshots) with just a little (but clearly visible) addition to the message like “ERROR: Test class.method(params) failed but WILL BE RETRIED”

[juherr]

@rschmitt Is #698 fixing this issue or do you want to keep it open for any reason?