JDK17 broken test scanning

When running a JDK11-based project on JDK17, it fails with odd messages like below. This project primarily uses TestNG but has some JUnit tests for external tooling, e.g. to leverage Guava’s collection tests, osgi testing, and similar. Generally these tests can live together and automagic scanning picks up what is appropriate.

In the stacktrace below, CacheSpec is a custom TestNG annotation to influence generation of parameterized test arguments. The nested class Loader is an enum for the argument options, e.g. @CacheSpec(removalListener = MOCKITO). For some reason only on JDK17 does the junit runner try to instantiate the enum as if a regular class.

Full logs using 4.13.2

com.github.benmanes.caffeine.cache.testing.CacheSpec$Loader > initializationError FAILED
    org.junit.runners.model.InvalidTestClassError: Invalid test class 'com.github.benmanes.caffeine.cache.testing.CacheSpec$Loader':
      1. Test class should have exactly one public constructor
      2. No runnable methods
        at org.junit.runners.ParentRunner.validate(ParentRunner.java:525)
        at org.junit.runners.ParentRunner.<init>(ParentRunner.java:102)
        at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:84)
        at org.junit.runners.JUnit4.<init>(JUnit4.java:23)
        at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
        at org.junit.internal.builders.AllDefault PossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:37)
        at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
        at org.junit.internal.requests.ClassRequest.createRunner(ClassRequest.java:28)
        at org.junit.internal.requests.MemoizingRequest.getRunner(MemoizingRequest.java:19)

Switching to JUnit 5 with the vintage engine fails after 6 minutes with OOME before running the tests.

Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
	at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)
	at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:175)
	at java.base/java.lang.reflect.Method.specificToStringHeader(Method.java:421)
	at java.base/java.lang.reflect.Executable.sharedToString(Executable.java:115)
	at java.base/java.lang.reflect.Method.toString(Method.java:412)
	at org.junit.platform.commons.util.ReflectionUtils.defaultMethodSorter(ReflectionUtils.java:1569)
	at org.junit.platform.commons.util.ReflectionUtils$$Lambda$140/0x0000000800d2a030.compare(Unknown Source)
	at java.base/java.util.TimSort.binarySort(TimSort.java:296)
	at java.base/java.util.TimSort.sort(TimSort.java:239)
	at java.base/java.util.Arrays.sort(Arrays.java:1307)
	at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:353)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.junit.platform.commons.util.ReflectionUtils.toSortedMutableList(ReflectionUtils.java:1546)
	at org.junit.platform.commons.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:1492)
	at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1433)
	at org.junit.platform.commons.util.ReflectionUtils.getSuperclassMethods(ReflectionUtils.java:1639)
	at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1436)
	at org.junit.platform.commons.util.ReflectionUtils.getSuperclassMethods(ReflectionUtils.java:1639)
	at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1436)
	at org.junit.platform.commons.util.ReflectionUtils.getSuperclassMethods(ReflectionUtils.java:1639)
	at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1436)
	at org.junit.platform.commons.util.ReflectionUtils.getSuperclassMethods(ReflectionUtils.java:1639)
	at org.junit.platform.commons.util.ReflectionUtils.findAllMethodsInHierarchy(ReflectionUtils.java:1436)
	at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1417)
	at org.junit.platform.commons.util.ReflectionUtils.findMethods(ReflectionUtils.java:1403)
	at org.junit.vintage.engine.descriptor.TestSourceProvider.lambda$findMethod$1(TestSourceProvider.java:75)

[kcooney]

I’m not sure when Marc (or anyone else who helps out with JUnit) will find cycles to track this down, but I did want to point out that JUnit 4.x doesn’t come bundled with a classpath scanning or other test-discovery code.

This might be an issue with Gradle or a Gradle plugin.

[ben-manes]

I’ll try to reproduce the OOME on JUnit5 after Gradle has official JDK17 support, just to avoid the noise if not your team’s responsibility.