Non-adjacent repeatable annotations do not work with @CartesianProductTest

While working on #414, I realized there is a nasty issue when repeatable annotations have other annotations (repeatable or not) in between.

Take this example:

@CartesianProductTest
@IntRangeSource(from = 0, to = 4)
@CartesianValueSource(longs = { 2, 4 })
@IntRangeSource(from = 0, to = 4)
void test(int i1, long l, int i2) {
}

The test execution will fail with:

org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter [long arg1] in method [void org.example.MyTest.test(int,long,int)].
	at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameter(ExecutableInvoker.java:200)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameters(ExecutableInvoker.java:183)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameters(ExecutableInvoker.java:144)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:96)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:171)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

This is due to the fact that the absolute declaration order of repeatable annotations is not kept when accessing them via reflection, as each group is encapsulated in the container annotation (still keeping the relative order, though).

So, when the extension invokes the corresponding argument providers, the order of:

@IntRangeSource(from = 0, to = 4)
@CartesianValueSource(longs = { 2, 4 })
@IntRangeSource(from = 0, to = 4)

is actually:

@IntRangeSources({
    @IntRangeSource(from = 0, to = 4)
    @IntRangeSource(from = 0, to = 4)
})
@CartesianValueSource(longs = { 2, 4 })

and an int value is proposed to the second parameter, which is a long, hence the exception.

At the moment I don’t see any way to solve this, but I’m far from being an expert on this topic.

What I see is that the error message does not help an end-user who is not fully familiar with how repeatable annotation ordering works under the hood (like me 🙂). He will not understand that reordering the annotations, and the corresponding parameters, is needed to make the test work.

I think the docs should be enhanced to mention this limitation. Changing annotations and parameters order should not be an issue for the users.

Another option could be to make the extension able to detect those cases and fail with a better message, but I’m not sure it is possible.

Any thoughts?

[Michael1993]

I’ve been thinking about this. The way I see it, we have 3 options: the two @scordio already listed and to try and make the extension work with the annotation mismatch.

The documentation update should be fairly straightforward.

I have a hunch how the extension could detect such a scenario, but it would require more reflection and evaluation (increased complexity), which could make the extension both slower and harder to maintain, which could bring unintended downsides with it.

Right now, I’m leaning towards updating the documentation and opening an issue which people can “upvote” if they want to see this feature implemented… Getting some more feedback would be great.

[Michael1993]

The new recommendation for anyone encountering this issue is: try CartesianTest instead of CartesianProductTest!

(…and if you still have trouble, feel free to open a new issue.)