Netty on GraalVM with substitutions not fully working as expected

Expected behavior

The NetUtilSubstitutions implement a set InetAddress, Inet4Address and Inet6Address dummy implementation.

Actual behavior

See logging with exceptions below. I expect no exceptions related to netty.

Steps to reproduce

Have a running docker engine since it uses the buildpacks from spring.

git clone [e-mail]:ArjanSchouten/spring-native-netty-reproduce.git
cd spring-native-netty-reproduce
./gradlew bootBuildImage

Minimal yet complete reproducer code (or URL to code)

See steps to reproduce.

Netty version

4.1.54.Final

JVM version (e.g. java -version)

11.0

OS version (e.g. uname -a)

Darwin computername 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64

In PR https://github.com/netty/netty/pull/10630 the NetUtilSubstitutions where added. I try to use Netty with https://github.com/spring-projects-experimental/spring-graalvm-native and it fails with:

[creator]     22:25:44.836 [ForkJoinPool-2-worker-15] DEBUG io.netty.buffer.AdvancedLeakAwareByteBuf - -Dio.netty.leakDetection.acquireAndReleaseOnly: false
    [creator]     Warning: class initialization of class io.netty.util.internal.logging.Log4JLogger failed with exception java.lang.NoClassDefFoundError: org/apache/log4j/Priority. This class will be initialized at run time because option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=io.netty.util.internal.logging.Log4JLogger to explicitly request delayed initialization of this class.
    [creator]     22:25:49.486 [ForkJoinPool-2-worker-3] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
    [creator]     22:25:49.486 [ForkJoinPool-2-worker-3] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
    [creator]     22:25:49.605 [ForkJoinPool-2-worker-7] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacityPerThread: 4096
    [creator]     22:25:49.606 [ForkJoinPool-2-worker-7] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxSharedCapacityFactor: 2
    [creator]     22:25:49.606 [ForkJoinPool-2-worker-7] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.linkCapacity: 16
    [creator]     22:25:49.606 [ForkJoinPool-2-worker-7] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.ratio: 8
    [creator]     22:25:49.607 [ForkJoinPool-2-worker-7] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.delayedQueue.ratio: 8
    [creator]     22:25:54.431 [ForkJoinPool-2-worker-11] DEBUG io.netty.handler.codec.compression.ZlibCodecFactory - -Dio.netty.noJdkZlibDecoder: false
    [creator]     22:25:54.432 [ForkJoinPool-2-worker-11] DEBUG io.netty.handler.codec.compression.ZlibCodecFactory - -Dio.netty.noJdkZlibEncoder: false
    [creator]     22:25:55.835 [ForkJoinPool-2-worker-9] DEBUG io.netty.handler.ssl.OpenSsl - netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
    [creator]     22:25:57.117 [ForkJoinPool-2-worker-7] DEBUG io.netty.channel.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 16
    [creator]     22:25:58.173 [ForkJoinPool-2-worker-15] DEBUG io.netty.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
    [creator]     [/layers/paketo-buildpacks_spring-boot-native-image/native-image/com.example.springnative.Application:169]     analysis:  42,689.86 ms,  2.63 GB
    [creator]     Fatal error:com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing io.netty.util.NetUtil.<clinit>()
    [creator]     Parsing context: <no parsing context available>
    [creator]
    [creator]           at com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:138)
    [creator]           at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:327)
    [creator]           at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureParsed(MethodTypeFlow.java:302)
    [creator]           at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:103)
    [creator]           at com.oracle.graal.pointsto.BigBang$1.run(BigBang.java:428)
    [creator]           at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
    [creator]           at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
    [creator]           at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    [creator]           at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    [creator]           at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    [creator]           at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    [creator]           at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
    [creator]     Caused by: org.graalvm.compiler.java.BytecodeParser$BytecodeParserError: com.oracle.svm.core.util.VMError$HostedError: Error in @InjectAccessors handling of field io.netty.util.NetUtil.LOCALHOST4, accessors class io.netty.util.NetUtilSubstitutions$NetUtilLocalhost4Accessor: found no method named set or setLOCALHOST4
    [creator]           at parsing io.netty.util.NetUtil.<clinit>(NetUtil.java:139)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.throwParserError(BytecodeParser.java:2580)
    [creator]           at com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.throwParserError(SharedGraphBuilderPhase.java:100)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3418)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3220)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1090)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:984)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:84)
    [creator]           at com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:74)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:214)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
    [creator]           at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:223)
    [creator]           at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:357)
    [creator]           at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:313)
    [creator]           ... 10 more
    [creator]     Caused by: com.oracle.svm.core.util.VMError$HostedError: Error in @InjectAccessors handling of field io.netty.util.NetUtil.LOCALHOST4, accessors class io.netty.util.NetUtilSubstitutions$NetUtilLocalhost4Accessor: found no method named set or setLOCALHOST4
    [creator]           at com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:68)
    [creator]           at com.oracle.svm.hosted.phases.InjectedAccessorsPlugin.error(InjectedAccessorsPlugin.java:152)
    [creator]           at com.oracle.svm.hosted.phases.InjectedAccessorsPlugin.handleField(InjectedAccessorsPlugin.java:89)
    [creator]           at com.oracle.svm.hosted.phases.InjectedAccessorsPlugin.handleStoreStaticField(InjectedAccessorsPlugin.java:62)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genPutStatic(BytecodeParser.java:4970)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genPutStatic(BytecodeParser.java:4941)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5335)
    [creator]           at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3413)
    [creator]           ... 22 more
    [creator]     Error: Image build request failed with exit status 1

I tried to add a substitute myself but it looks like there can only be one substitute. It looks like the author of the stackoverflow answer posted the PR: https://stackoverflow.com/questions/63328298/how-do-you-debug-a-no-instances-of-are-allowed-in-the-image-heap-when-buil (which is great!!)

I think that a set implementation like the stackoverflow answer from @rpuch like in part 4 of the answer will solve my issue?! @rpuch can you validate that my assumption is correct?

_edit: according to the documentation: https://javadoc.io/doc/org.graalvm.nativeimage/svm/latest/com/oracle/svm/core/annotate/InjectAccessors.html_

Inject accessors methods for the field denoted using a Alias annotation. All loads and stores to the original field are redirected to accessor methods located in the class provided in the InjectAccessors.value() property. The class must implement the marker interface InjectAccessors. The accessor methods are static methods in that class, named either get / set or getFoo / setFoo for a field name foo. Depending on the kind of accessor (get / set for a static / non-static field), the accessor must have 0, 1, or 2 parameters. If the field is non-static, the first method parameter is the accessed object. The type of the parameter must be the class that declared the field. The null check on the object is performed before the accessor is called, in the same way as the null check for a regular field access. For get-accessors, the return type of the method must be the type of the field. For set-accessors, the last method parameter must be the type of the field and denotes the value stored to the field. If no set-accessor is provided, stores to the field lead to a fatal error during image generation. If no get-accessor is provided, loads of the field lead to a fatal error during image generation. The injected accessors must not access the original field. Since all field accesses use the accessors, that would lead to a recursive call of the accessors. Instead, data must be stored in either a new static field, or an injected instance field.

According to the documentation, is the setLOCALHOST4 accessed illegally or do we have to implement the static set method as an empty method?

(I believe this will be a blocker for https://github.com/spring-projects-experimental/spring-graalvm-native/issues/184 as well)

[vjovanov]

The GraalVM Native Image team is preparing a PR that will revert build-time initialization for Netty. By aggregating all the issues and request that we get due to this feature, it is clear to us that the default setting should be run time (the same as GraalVM): the maintenance overhead far outweighs the startup performance benefits.

Projects that want to keep build-time initialization for netty can always take the existing config and use it in their projects, or make a separate repo. Of course, all issues related to build-time initialization from GraalVM shall be redirected to those project or that repo.

[sdeleuze]

This is embarassing, but with #10630 I broke Netty for the cases when NetUtil is initialized at run-time.

FYI this change broke native Netty support in Spring Boot 2.4.0. I will adapt our support accordingly.

@ArjanSchouten could you please check Netty built from #10799 to see whether it fixes the problem for you?

@rpuch Thanks for working on a fix, but looking at the changes in #10630, I understand the problem you are trying to solve but maybe we should take a step back on why those changes are needed.

@normanmaurer Just to give more context, as you know GraalVM native started its life with build time initialization by default, but since is clearly moving towards runtime initialization by default to favor compatibility, avoid to have 2 distinct behaviors of Java to support in the ecosystem, and because build time init is really required in just a few places where it is critical for performances or footprint:

• GraalVM native 19.0.0 switched to runtime init by default
• GraalVM native 20.2.0 moved most of the JDK to runtime init
• I think GraalVM and Project Leyden teams are trying to limit the differences of behavior in order to avoid to have 2 different behaviors of Java, in order to avoid a maintainability nightmare well illustrated by those changes
• I think @vjovanov from GraalVM team is about to provide a PR to restore by default runtime init by default in Netty.

In a nustshell, my proposal is to:

• Rollback #10630 changes
• Merge @vjovanov upcoming PR in order to restore defaults consistent with GraalVM one, with more consistency between JVM and native
• Evolve Netty native support with a goal of limiting substitutions, favor consistency about JVM and native flavors, make Netty consistent with other libraries that are by default initialized at runtime in GraalVM native
• Use build time init if needed on the few classes where there are needed to perform build time code removal or when critical for performances
• Document with @rpuch and others interested how to use Netty with build time init and sync with them in order to allow them to adapt to those changes