RepeatFailedTest causes sporadically PreconditionViolationException

A failed test repeated through @RepeatFailedTest sporadically leads to:

org.junit.platform.commons.PreconditionViolationException: Illegal state: required test method is not present in the current ExtensionContext

Link to a failed build: https://travis-ci.org/github/spring-projects/spring-data-mongodb/jobs/678085125

Reproducer: https://github.com/spring-projects/spring-data-mongodb/blob/46ab6b4c945fa5eef1acc000c9d9735a715c8f31/spring-data-mongodb/src/test/java/org/springframework/data/mongodb/core/messaging/ChangeStreamTests.java#L201

[nipafx]

What’s going on?

The @RepeatFailedTest extension is mainly a test template, which means it becomes it’s own container for the tests it generates and doesn’t need an additional @Test annotation (just like @ParameterizedTest).

In the linked example, the test method is annotated with @Test and @RepeatFailedTest, though, just like this example:

@Test
@RepeatFailedTest(3)
void failsNever() { }

This means if the test is successful, it’s executed twice:

If it isn’t successful, things get a bit more complicated. I said, this is extension is mainly a test template. It’s also a TestExecutionExceptionHandler, so it can catch exceptions and trigger an additional run:

@Override
public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
	// this `context` (M) is a child of the context passed to `provideTestTemplateInvocationContexts` (T),
	// which means M's store content is invisible to T's store; this can be fixed by using T's store here
	ExtensionContext templateContext = context
			.getParent()
			.orElseThrow(() -> new IllegalStateException(
				"Extension context \"" + context + "\" should have a parent context."));
	repeaterFor(templateContext).failed(throwable);
}

private static FailedTestRepeater repeaterFor(ExtensionContext context) {
	Method repeatedTest = context.getRequiredTestMethod();
	return context
			.getStore(NAMESPACE)
			.getOrComputeIfAbsent(repeatedTest.toString(), __ -> FailedTestRepeater.createFor(repeatedTest),
				FailedTestRepeater.class);
}

Note the lengthy comment (ha!), which explains why we go from the failed test’s context to its parent (in handleTestExecutionException) to access the store (in repeaterFor). The implicit assumption is that handleTestExecutionException is called for a failed test that was generated from the annotated method (hence “go to parent!” ~> “get store”).

Now, if the method is also annotated with @Test and it fails, this assumption is invalid: The extension is still a TestExecutionExceptionHandler and thus is called to handle that failed test, but since that test is the method itself (not a generated test), it’s parent is a test class (not the test method) and thus getRequiredTestMethod in repeaterFor fails with the observed PreconditionViolationException.

What can the user do?

Remove @Test. This is not optional because even if the extension would detect this case and rethrow the exception, the @Test-test would fail and thus break the build (regardless of what the @RepeatFailedTest-tests do).

What can we do?

Two things come to mind:

• improve documentation to clarify that @Test is not needed
• add code that detects the case and provides a helpful error message

[mp911de]

No strong opinion on naming. It probably doesn’t make sense to rename and cause downstream effort as current proposal are mostly different but not significant more comprehensive.