Upgrade to JC Tools 3.0 changes queue behavior when compiled natively on GraalVM.

The upgrade to JC Tools 3.0 from 4.1.45 to 4.1.46 changes queue behavior when compiled natively on GraalVM.

For Netty 4.1.45 we used the following substitution in the Neo4j Java Driver

@TargetClass(className = "io.netty.channel.nio.NioEventLoop")
 final class Target_io_netty_channel_nio_NioEventLoop {

     @Substitute
     private static Queue<Runnable> newTaskQueue0(int maxPendingTasks) {
         return new LinkedBlockingDeque<>();
     }
 }

to force Netty to not use JC Tools Mscp queues. This is the same substitution that Quarkus uses (see https://github.com/quarkusio/quarkus/blob/0513ad94fd93f4ac377b796521a03dd3a52a156b/extensions/netty/runtime/src/main/java/io/quarkus/netty/runtime/graal/NettySubstitutions.java#L313-L320) and which I proposed to include in Netty itself here (https://github.com/netty/netty/pull/9989).

This substitution is not enough anymore since Netty 4.1.46.

Netty 4.1.46 updated JC Tools to 3.0. When I downgrade this to 2.1.1 again, the substitution works as expected.

I have the suspicion that this change here https://github.com/JCTools/JCTools/commit/2ac1a663b3a378225ea1269e25b6383751954c4c triggers a different behavior in native image mode.

This can be mitigated by allowing unsafe, reflective field access to a couple of classes and fields of JC Tools in a reflection-config.json.

 {
    "name" : "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerIndexField",
    "fields": [
      {"name": "producerIndex", "allowUnsafeAccess": true}
    ]
  },
  {
    "name" : "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerLimitField",
    "fields": [
      {"name": "producerLimit", "allowUnsafeAccess": true}
    ]
  },
  {
    "name" : "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueConsumerIndexField",
    "fields": [
      {"name": "consumerIndex", "allowUnsafeAccess": true}
    ]
  },
  {
    "name" : "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields",
    "fields": [
      {"name": "producerIndex", "allowUnsafeAccess": true}
    ]
  },
  {
    "name" : "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueColdProducerFields",
    "fields": [
      {"name": "producerLimit", "allowUnsafeAccess": true}
    ]
  },
  {
    "name" : "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueConsumerFields",
    "fields": [
      {"name": "consumerIndex", "allowUnsafeAccess": true}
    ]
  }

If this is not done, any thing bootstrapping Netty will fail with something in this area:

Exception in thread "main" java.lang.ExceptionInInitializerError
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:290)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:235)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:235)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:235)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:235)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:235)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:235)
	at java.lang.Class.ensureInitialized(DynamicHub.java:499)
	at io.netty.util.internal.PlatformDependent$Mpsc.newMpscQueue(PlatformDependent.java:934)
	at io.netty.util.internal.PlatformDependent.newMpscQueue(PlatformDependent.java:945)
	at io.netty.channel.nio.NioEventLoop.newTaskQueue0(NioEventLoop.java:279)
	at io.netty.channel.nio.NioEventLoop.newTaskQueue(NioEventLoop.java:150)
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:138)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:96)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:91)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44)
	at ac.simons.netty.example.Client.main(Client.java:21)
Caused by: java.lang.RuntimeException: java.lang.NoSuchFieldException: producerIndex
	at io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess.fieldOffset(UnsafeAccess.java:111)
	at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields.<clinit>(BaseMpscLinkedArrayQueue.java:41)
	at com.oracle.svm.core.hub.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:350)
	at com.oracle.svm.core.hub.ClassInitializationInfo.initialize(ClassInitializationInfo.java:270)
	... 29 more
Caused by: java.lang.NoSuchFieldException: producerIndex
	at java.lang.Class.getDeclaredField(DynamicHub.java:2411)
	at io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess.fieldOffset(UnsafeAccess.java:107)
	... 32 more

I have created the most simple reproducer I could: https://github.com/michael-simons/native-netty-timeserver-and-client Which is basically the vanilla server/client code from the Time server example of yours.

It can be compiled and run natively on the GraalVM. If you want to see the error, take this commit and update to anything >= 4.1.46 and stuff will fail.

The next commit contains the fixes.

Expected behavior or outcome

Nothing concrete at the moment: We at Neo4j can work around this, but it might have an impact on other projects that use Netty but are aiming for native compilation.

The substation mechanism seems brittle to say at least. Under the light of this experience here, I’d rather think about closing my pr but suggest picking up the suggested reflection config above for a future version of Netty. I personally don’t feel to well to refer to shaded things (which we shade again in our driver).

As this was discovered by the kind Quarkus team here https://github.com/quarkusio/quarkus/pull/10200, I’m also paging @gsmet and @cescoffier as they might run into the same issues with the Netty substitutions in Quarkus itself.

Maybe @nitsanw has an idea why the movement of the the static field initializations are now triggered early?

[cescoffier]

… we already using 4.1.49, so it needs to be updated.

[jvican]

FWIW it looks like -Dio.netty.noUnsafe is not necessary to get a performant Netty-based application compiled by GraalVM. I’ve been using this flag in production and recently discovered that I could remove it and Netty would behave just fine.