Error transforming FacesServlet

Our Jetty 9.4 tests started failing recently, apparently following the latest 9.4 release. Reproducing it locally, I got the following retransformation-related error:

[jetty] 2020-11-23 06:33:15,578 [Attach Listener] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Error invoking java.lang.instrument.Instrumentation#retransformClasses
[jetty] java.lang.IllegalStateException: Error invoking java.lang.instrument.Instrumentation#retransformClasses
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(AgentBuilder.java:6916) ~[?:?]
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:7187) ~[?:?]
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:7032) [?:?]
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:4868) [?:?]
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:9502) [?:?]
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:9423) [?:?]
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(AgentBuilder.java:11025) [?:?]
[jetty] at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:261) [?:?]
[jetty] at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:164) [?:?]
[jetty] at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:150) [?:?]
[jetty] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
[jetty] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) ~[?:?]
[jetty] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
[jetty] at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
[jetty] at co.elastic.apm.agent.bci.AgentMain.init(AgentMain.java:119) [elastic-apm-agent-b41211a1ba26a466d8245d6f0fdbe7fa.jar:?]
[jetty] at co.elastic.apm.agent.bci.AgentMain.agentmain(AgentMain.java:69) [elastic-apm-agent-b41211a1ba26a466d8245d6f0fdbe7fa.jar:?]
[jetty] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
[jetty] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) ~[?:?]
[jetty] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
[jetty] at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
[jetty] at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513) [?:?]
[jetty] at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535) [?:?]
[jetty] Caused by: java.lang.InternalError
[jetty] at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method) ~[?:?]
[jetty] at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167) ~[?:?]
[jetty] at jdk.internal.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) ~[?:?]
[jetty] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
[jetty] at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
[jetty] at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(AgentBuilder.java:6908) ~[?:?]
[jetty] ... 21 more

After some investigation, it seemed that this error occurs in the attempt to retransform the javax.faces.webapp.FacesServlet class that comes from the myfaces 2.2.12 jar. Upgrading to 2.3.2 seems to solve the issue, so I am not digging further into it at the moment, although the retransformation error remains.

The original and transformed bytecode can be found in FacesServlet_bytecode.zip. A quick look didn’t reveal anything to me. We are applying this advice as well as these two.

I hope this provides enough info in order to find the problem.

[raphw]

I’d say it’s worth looking into, especially since you can provide a reproduction. But I expect they’ll say that the class file upgrade is unintended use.

[raphw]

Likely this is related to the increment in the class file version. There is nothing particularly different in the two versions besides the first one being compiled with Java 6 and the other one with Java 8. I assume that if you skipped your patch to 7, it would work.

This besides the class file upgrade being allowed. I think something interfers with the validation check when comparing the class file. The compilation results are still odd, the wide bytecodes disappeared in the new version however. I am guessing now but it’s probably the strangeness of the Eclipse Java compiler I am seeing here. I am just guessing but I think the removal of the wide byte code through ASM fails some transformation compatibility check. I will try to reconstruct a few scenarios next week, but if I am write there is little I can do (but patch the verifier in OpenJDK if I can pin it down to that, and then the question would be if this got backported anyways).