Unable to load instrumentation a lot of modules after update version

Describe the bug after upgrade version v1.9.1 to v1.12.1, I’ve been facing below errors at start of application. how can I investigate further?

...
[otel.javaagent 2022-04-27 19:55:21:225 +0900] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation reactor [class io.op
entelemetry.javaagent.instrumentation.reactor.operator.ContextPropagationOperatorInstrumentationModule]
[otel.javaagent 2022-04-27 19:55:21:227 +0900] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation opentelemetry-api [c
lass io.opentelemetry.javaagent.instrumentation.opentelemetryapi.OpenTelemetryApiInstrumentationModule]
[otel.javaagent 2022-04-27 19:55:21:228 +0900] [main] DEBUG io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Loading instrumentation executor [class io.o
pentelemetry.javaagent.instrumentation.javaconcurrent.ExecutorInstrumentationModule]
[otel.javaagent 2022-04-27 19:55:21:237 +0900] [main] ERROR io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader - Unable to load instrumentation executor [cla
ss io.opentelemetry.javaagent.instrumentation.javaconcurrent.ExecutorInstrumentationModule]
java.lang.IllegalArgumentException
        at org.objectweb.asm.ClassVisitor.<init>(Unknown Source)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor.<init>(TypeWriter.java:2322)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ValidatingClassVisitor.of(TypeWriter.java:2333)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForCreation.create(TypeWriter.java:5890)
        at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2213)
        at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:232)
        at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:204)
        at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3661)
        at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$Delegator.make(DynamicType.java:3899)
        at io.opentelemetry.javaagent.tooling.field.FieldAccessorInterfacesGenerator.makeFieldAccessorInterface(FieldAccessorInterfacesGenerator.java:72)
        at io.opentelemetry.javaagent.tooling.field.FieldAccessorInterfacesGenerator.generateFieldAccessorInterfaces(FieldAccessorInterfacesGenerator.java:35)
        at io.opentelemetry.javaagent.tooling.field.FieldBackedImplementationInstaller.<init>(FieldBackedImplementationInstaller.java:84)
        at io.opentelemetry.javaagent.tooling.field.VirtualFieldImplementationInstallerFactory.create(VirtualFieldImplementationInstallerFactory.java:37)
        at io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationModuleInstaller.install(InstrumentationModuleInstaller.java:88)
        at io.opentelemetry.javaagent.tooling.instrumentation.InstrumentationLoader.extend(InstrumentationLoader.java:34)
        at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:162)
        at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:95)
        at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:65)
        at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:35)
        at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:55)
        at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:44)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)

What did you expect to see? success of module loading

What version are you using? v1.12.1

Environment

[otel.javaagent 2022-04-27 19:55:19:715 +0900] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.12.1
[otel.javaagent 2022-04-27 19:55:19:716 +0900] [main] DEBUG io.opentelemetry.javaagent.tooling.VersionLogger - Running on Java 11.0.14. JVM OpenJDK 64-Bit Server VM - Amazon.com I
nc. - 11.0.14+9-LTS
[otel.javaagent 2022-04-27 19:55:19:716 +0900] [main] DEBUG io.opentelemetry.javaagent.tooling.AgentInstaller - io.opentelemetry.javaagent.tooling.AgentInstaller loaded on io.open
telemetry.javaagent.bootstrap.AgentClassLoader@5315b42e
[otel.javaagent 2022-04-27 19:55:19:892 +0900] [main] DEBUG io.opentelemetry.sdk.internal.JavaVersionSpecific - Using the APIs optimized for: Java 9+
[otel.javaagent 2022-04-27 19:55:20:593 +0900] [main] DEBUG io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder - Global OpenTelemetrySdk set to OpenTelemetry
Sdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.9.0, attributes={containe
r.id="b232a2865daee190f85e1f14f5437d8a674a6f733d5cb3af4112fd53791d61ca", deployment.environment="xxxxx", host.arch="amd64", host.name="xxxx", os.description="Linux 5.10.102", os.type="linux", process.command_line="/usr/lib/jvm/java-11-amazon-corretto:bin:java -Dcom.sun.management.jmxremote -Dcom.sun.manage
ment.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Dcom.sun.m
anagement.jmxremote.ssl=false -Dfile.encoding=UTF-8 -Djava.rmi.server.hostname=localhost -Dspring.config.location=classpath:/,file:/opt/xxxx/config/ -Dspring.profiles.active=feat
ure -XX:+HeapDumpOnOutOfMemoryError -XX:+UseContainerSupport -XX:HeapDumpPath=/opt/xxxx/log/java_oom_heap_dump_${POD_NAME}.hprof -javaagent:/opt/xxxx/kube-agent.jar -javaagent:/
opt/xxxx/opentelemetry-java-agent.jar -Dotel.javaagent.debug=true", process.executable.path="/usr/lib/jvm/java-11-amazon-corretto:bin:java", process.pid=1, process.runtime.description="Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.14+9-LTS", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="11.0.14+9-LTS", service.name="xxxx", telemetry.auto.version="1.12.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.12.0"}}, spanLimitsSupplier=SpanLimitsValue{maxNumb
erOfAttributes=64, maxNumberOfEvents=64, maxNumberOfLinks=64, maxNumberOfAttributesPerEvent=128, maxNumberOfAttributesPerLink=128, maxAttributeValueLength=2147483647}, sampler=Alw
aysOnSampler, spanProcessor=MultiSpanProcessor{spanProcessorsStart=[io.opentelemetry.javaagent.tooling.AddThreadDetailsSpanProcessor@783a467b], spanProcessorsEnd=[BatchSpanProcess
or{spanExporter=io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter@272113c4, scheduleDelayNanos=5000000000, maxExportBatchSize=512, exporterTimeoutNanos=30000000000}, Simpl
eSpanProcessor{spanExporter=io.opentelemetry.exporter.logging.LoggingSpanExporter@73e9cf30}], spanProcessorsAll=[BatchSpanProcessor{spanExporter=io.opentelemetry.exporter.otlp.tra
ce.OtlpGrpcSpanExporter@272113c4, scheduleDelayNanos=5000000000, maxExportBatchSize=512, exporterTimeoutNanos=30000000000}, io.opentelemetry.javaagent.tooling.AddThreadDetailsSpan
Processor@783a467b, SimpleSpanProcessor{spanExporter=io.opentelemetry.exporter.logging.LoggingSpanExporter@73e9cf30}]}}} by autoconfiguration

Compiler: (e.g., “AdoptOpenJDK 11.0.6”) Amazon.com Inc. OpenJDK 64-Bit Server VM 11.0.14+9-LTS OS: (e.g., “Ubuntu 20.04”) Kubernetes, amazoncorretto:11.0.15-al2

[laurit]

I think we should try to fix this.

[trask]

@chronosa also, what is kube-agent.jar? could be that is adding unshaded asm to the bootstrap class loader