Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

NoClassDefFound with Tomcat + Security Manager

See original GitHub issue

When a security manager is configured in Tomcat (might also happen with other server/apps), the standard output is polluted with lots of NoClassDefFoundError errors.

How to reproduce:

  • grant agent every permission in conf/catalina.policy , for example:
    grant codeBase "file:/path/to-agent/elastic-apm-agent-1.29.1-SNAPSHOT.jar" {
  permission java.security.AllPermission;
};
  • start tomcat with bin/catalina.sh run -security

Agent version: 1.29.0 + same on main as of writing Tomcat version: 9.0.x

Setting warmup_byte_buddy=false does not makes a difference here. While the symptoms might be similar, the fix for #2509 does not cover this case, thus it might not be related but similar.

catalina.out output 
2022-03-17 10:31:22,251 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2022-03-17 10:31:22,290 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.29.1-SNAPSHOT.UNKNOWN as tomcat-application on Java 11.0.8 Runtime version: 11.0.8+10 VM version: 11.0.8+10 (AdoptOpenJDK) Linux 5.13.0-30-generic
2022-03-17 10:31:22,290 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - warmup_byte_buddy: 'false' (source: Java System Properties)
2022-03-17 10:31:22,290 [main] WARN  co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2022-03-17 10:31:23,417 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2022-03-17 10:31:23,450 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2022-02-03T16:50:25Z",  "build_sha": "6afb387ae002e5a8dc2427b53225aa42681e4208",  "publish_ready": true,  "version": "8.0.0"}
17-Mar-2022 10:31:23.766 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.38
17-Mar-2022 10:31:23.769 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Sep 10 2020 08:20:30 UTC
17-Mar-2022 10:31:23.769 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.38.0
17-Mar-2022 10:31:23.769 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
17-Mar-2022 10:31:23.769 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.13.0-30-generic
17-Mar-2022 10:31:23.769 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
17-Mar-2022 10:31:23.769 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /home/sylvain/.sdkman/candidates/java/11.0.8.hs-adpt
17-Mar-2022 10:31:23.769 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.8+10
17-Mar-2022 10:31:23.770 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            AdoptOpenJDK
17-Mar-2022 10:31:23.770 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /home/sylvain/tools/tomcat/apache-tomcat-9.0.38
17-Mar-2022 10:31:23.770 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /home/sylvain/tools/tomcat/apache-tomcat-9.0.38
17-Mar-2022 10:31:23.770 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
17-Mar-2022 10:31:23.770 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
17-Mar-2022 10:31:23.770 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
17-Mar-2022 10:31:23.770 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/sylvain/tools/tomcat/apache-tomcat-9.0.38/conf/logging.properties
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -javaagent:/home/sylvain/dev/apm-agent-java_security-manager-warn/elastic-apm-agent/target/elastic-apm-agent-1.29.1-SNAPSHOT.jar
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Delastic.apm.warmup_byte_buddy=false
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.manager
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.policy==/home/sylvain/tools/tomcat/apache-tomcat-9.0.38/conf/catalina.policy
17-Mar-2022 10:31:23.771 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/sylvain/tools/tomcat/apache-tomcat-9.0.38
17-Mar-2022 10:31:23.772 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/sylvain/tools/tomcat/apache-tomcat-9.0.38
17-Mar-2022 10:31:23.772 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/sylvain/tools/tomcat/apache-tomcat-9.0.38/temp
17-Mar-2022 10:31:23.773 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
17-Mar-2022 10:31:24.011 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
17-Mar-2022 10:31:24.057 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [471] milliseconds
17-Mar-2022 10:31:24.115 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
17-Mar-2022 10:31:24.116 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.38]
17-Mar-2022 10:31:24.124 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/home/sylvain/tools/tomcat/apache-tomcat-9.0.38/webapps/docs]
java.lang.NoClassDefFoundError: co/elastic/apm/agent/servlet/ServletServiceNameHelper
  at co.elastic.apm.agent.servlet.servicename.ServletContainerInitializerServiceNameInstrumentation$JavaxInitServiceNameInstrumentation$AdviceClass.onEnter(ServletContainerInitializerServiceNameInstrumentation.java:89)
  at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:60)
  at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5128)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
  at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
  at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:129)
  at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:150)
  at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:140)
  at java.base/java.security.AccessController.doPrivileged(Native Method)
  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:688)
  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
  at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1133)
  at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1866)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
  at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
  at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1045)
  at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:429)
  at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1576)
  at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
  at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
  at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
  at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
  at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936)
  at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:843)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
  at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
  at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
  at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
  at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
  at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
  at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
  at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
  at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342)
  at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
Caused by: java.lang.ClassNotFoundException: co.elastic.apm.agent.servlet.ServletServiceNameHelper
  at net.bytebuddy.dynamic.loading.ByteArrayClassLoader.findClass(ByteArrayClassLoader.java:403)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
  at net.bytebuddy.dynamic.loading.ByteArrayClassLoader$ChildFirst.loadClass(ByteArrayClassLoader.java:1265)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
  ... 46 more

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
SylvainJugecommented, Mar 23, 2022
java.lang.RuntimeException: java.lang.UnsupportedOperationException: Could not access Unsafe class: sun.misc.Unsafe.defineClass(java.lang.String,[B,int,int,java.lang.ClassLoader,java.security.ProtectionDomain)
	at co.elastic.apm.agent.bci.IndyBootstrap.getIndyBootstrapMethod(IndyBootstrap.java:225) ~[elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.ElasticApmAgent.getTransformer(ElasticApmAgent.java:445) ~[elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.ElasticApmAgent.applyAdvice(ElasticApmAgent.java:405) ~[elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.ElasticApmAgent.initAgentBuilder(ElasticApmAgent.java:334) [elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:271) [elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:163) [elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:149) [elastic-apm-agent.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:156) [elastic-apm-agent.jar:1.29.0]
	at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:97) [elastic-apm-agent.jar:1.29.0]
	at co.elastic.apm.agent.premain.AgentMain.premain(AgentMain.java:49) [elastic-apm-agent.jar:1.29.0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491) [?:?]
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503) [?:?]
Caused by: java.lang.UnsupportedOperationException: Could not access Unsafe class: sun.misc.Unsafe.defineClass(java.lang.String,[B,int,int,java.lang.ClassLoader,java.security.ProtectionDomain)
	at net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$Unavailable.initialize(ClassInjector.java:2086) ~[elastic-apm-agent.jar:?]
	at net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe.injectRaw(ClassInjector.java:1813) ~[elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.IndyBootstrap.loadClassInBootstrap(IndyBootstrap.java:275) ~[elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.IndyBootstrap.initIndyBootstrap(IndyBootstrap.java:233) ~[elastic-apm-agent.jar:?]
	at co.elastic.apm.agent.bci.IndyBootstrap.getIndyBootstrapMethod(IndyBootstrap.java:219) ~[elastic-apm-agent.jar:?]
	... 19 more

We can now reproduce what you observed with tests added in https://github.com/elastic/apm-agent-java/pull/2539.

0reactions
eyalkorencommented, Nov 14, 2022

@SylvainJuge did you try to debug the BB bytecode lookup and compare it between with and without the security manager? Either we need to fix something in our DiscriminatingMultiParentClassLoader lookup logic or something in BB configuration that doesn’t work with our special resource packaging within the jar. My guess is that when security manager is used, there is some other lookup path that BB employs.

Read more comments on GitHub >

github_iconTop Results From Across the Web

ClassNotFoundException issued by URLClassLoader when ...
For an experimental project, I am trying to load a class with URLClassLoader with the security manager enabled. Despite the ...
Read more >
[REPOST]Tomcat with security manager + NoClassDefFoundError
Subject: Re: [REPOST]Tomcat with security manager + NoClassDefFoundError. Is there a part in your error message that says "Root Cause"? If so, what...
Read more >
48097 – NoClassDefFoundError on first access of first jsp
Its "development" option is set to true or false? Is your Tomcat version 6.0.20? Are you running with SecurityManager enabled?
Read more >
The required Server component failed to start so Tomcat is ...
After you upgrade to the latest release, Network Security Manager service ... of VMjava/lang/NoClassDefFoundError: java/lang/object Or: The ...
Read more >
ExceptionUtil with Tomcat Security Manager - Hibernate Forums
class cannot be loaded because access is denied. I see the NoClassDefFoundError: org/hibernate/exception/ExceptionUtils Exception instead of the ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Safely remove the enable_log_correlation config option

Next page

Slf4j java.lang.LinkageError