karaf error when using apm 1.3.0

Hallo folks,

sadly, I have to open a bug report regarding the APM 1.3.0. I sadly don’t know if it has worked with any other version before, but it seems to be an issue regarding “Apache Karaf”. Hopefully some of you can jump in and resolve this matter, so we can use the elastic jvm apm.

Describe the bug Nexus OSS 3.15.1 & 3.14.0 won’t start after adding APM 1.3.0 as a java option. I tried it with the UNIX and Mac version. Error Log can be found down further down.

java -version :( java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

To Reproduce Steps to reproduce the behaviour: When I write /nexus, the / refers to the top directory when the tar.gz is extracted.

1. Download nexus oss 3.15.1 from sonatype (https://www.sonatype.com/nexus-repository-oss)
2. Download elastic-apm-1.3.0.jar
3. Use default nexus oss config
4. Add the apm as javaagent in /nexus/bin/nexus.vmoptions
5. Start nexus with /nexus/bin/nexus start
6. See error from JVM in /sonatype-work/nexus3/log/jvm.log
7. See error from nexus in/sonatype-work/nexus3/log/nexus.log

Expected behaviour Even when adding elastic-apm-1.3.0.jar, nexus oss should start properly.

Debug logs Attach your debug logs. See the documentation about how to enable debug logging.

Click to expand

NEXUS

❱ cat ../../sonatype-work/nexus3/log/nexus.log
2019-01-21 21:23:25,830+0100 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.pax.logging.NexusLogActivator - start
2019-01-21 21:23:26,476+0100 ERROR [FelixDispatchQueue]  *SYSTEM org.apache.karaf.features.core - FrameworkEvent ERROR - org.apache.karaf.features.core
org.osgi.framework.BundleException: Activator start error in bundle org.apache.karaf.features.core [32].
	at org.apache.felix.framework.Felix.activateBundle(Felix.java:2288)
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2144)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1371)
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/sun/xml/internal/bind/DatatypeConverterImpl
	at org.apache.karaf.features.internal.model.Dependency_JaxbXducedAccessor_prerequisite.parse(TransducedAccessor_field_Boolean.java:48)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.StructureLoader.startElement(StructureLoader.java:195)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.ProxyLoader.startElement(ProxyLoader.java:45)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallingContext._startElement(UnmarshallingContext.java:559)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallingContext.startElement(UnmarshallingContext.java:538)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:60)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:153)
	at com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:229)
	at com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:266)
	at com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:235)
	at com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:266)
	at com.sun.xml.internal.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:235)
	at com.sun.xml.internal.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:112)
	at com.sun.xml.internal.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:95)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:356)
	at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:337)
	at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:127)
	at org.apache.karaf.features.internal.model.JaxbUtil.unmarshalValidate(JaxbUtil.java:133)
	at org.apache.karaf.features.internal.model.JaxbUtil.unmarshal(JaxbUtil.java:101)
	at org.apache.karaf.features.internal.service.RepositoryImpl.load(RepositoryImpl.java:94)
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.loadRepository(FeaturesServiceImpl.java:451)
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:467)
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.addRepository(FeaturesServiceImpl.java:462)
	at org.apache.karaf.features.internal.service.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:101)
	at org.apache.karaf.features.internal.service.BootFeaturesInstaller.start(BootFeaturesInstaller.java:90)
	at org.apache.karaf.features.internal.osgi.Activator.doStart(Activator.java:268)
	at org.apache.karaf.util.tracker.BaseActivator.start(BaseActivator.java:86)
	at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697)
	at org.apache.felix.framework.Felix.activateBundle(Felix.java:2238)
	... 4 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.sun.xml.internal.bind.DatatypeConverterImpl not found by org.apache.karaf.features.core [32]
	at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1550)
	at org.apache.felix.framework.BundleWiringImpl.access$200(BundleWiringImpl.java:79)
	at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1958)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 33 common frames omitted

JVM

cat ../../sonatype-work/nexus3/log/jvm.log
<?xml version='1.0' encoding='UTF-8'?>
<hotspot_log version='160 1' process='41761' time_ms='1548102202490'>
<vm_version>
<name>
Java HotSpot(TM) 64-Bit Server VM
</name>
<release>
25.181-b13
</release>
<info>
Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for bsd-amd64 JRE (1.8.0_181-b13), built on Jul  7 2018 01:02:31 by &quot;java_re&quot; with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
</info>
</vm_version>
<vm_arguments>
<args>
-Dinstall4j.jvmDir=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre -Dexe4j.moduleName=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/bin/nexus -XX:+UnlockDiagnosticVMOptions -Dinstall4j.launcherId=245 -Dinstall4j.swt=false -Di4jv=0 -Di4jv=0 -Di4jv=0 -Di4jv=0 -Di4jv=0 -Xms1200M -Xmx1200M -XX:MaxDirectMemorySize=2G -XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass -XX:+LogVMOutput -XX:LogFile=../sonatype-work/nexus3/log/jvm.log -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true -Dkaraf.home=. -Dkaraf.base=. -Dkaraf.etc=etc/karaf -Djava.util.logging.config.file=etc/karaf/java.util.logging.properties -Dkaraf.data=../sonatype-work/nexus3 -Djava.io.tmpdir=../sonatype-work/nexus3/tmp -Dkaraf.startLocalConsole=false -javaagent:/Users/pkahr/Downloads/elastic-apm-agent-1.3.0.jar -Di4j.vpt=true
</args>
<command>
com.install4j.runtime.launcher.UnixLauncher start 9d17dc87   org.sonatype.nexus.karaf.NexusMain
</command>
<launcher>
SUN_STANDARD
</launcher>
<properties>
java.vm.specification.name=Java Virtual Machine Specification
java.vm.version=25.181-b13
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.vm.info=mixed mode, sharing
java.ext.dirs=/Users/pkahr/Library/Java/Extensions:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/ext:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
java.endorsed.dirs=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/endorsed
sun.boot.library.path=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib
java.library.path=/Users/pkahr/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
java.home=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre
java.class.path=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/i4jruntime.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/lib/boot/nexus-main.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/lib/boot/org.apache.karaf.main-4.0.9.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/lib/boot/org.osgi.core-6.0.0.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/lib/boot/org.apache.karaf.diagnostic.boot-4.0.9.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/lib/boot/org.apache.karaf.jaas.boot-4.0.9.jar
sun.boot.class.path=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/resources.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/rt.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/sunrsasign.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/jsse.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/jce.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/charsets.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/lib/jfr.jar:/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre/classes
java.vm.specification.vendor=Oracle Corporation
java.vm.specification.version=1.8
java.vm.vendor=Oracle Corporation
install4j.jvmDir=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/.install4j/jre.bundle/Contents/Home/jre
exe4j.moduleName=/Users/pkahr/Downloads/nexus-3.15.1-01-mac/nexus-3.15.1-01/bin/nexus
install4j.launcherId=245
install4j.swt=false
i4jv=0
java.net.preferIPv4Stack=true
karaf.home=.
karaf.base=.
karaf.etc=etc/karaf
java.util.logging.config.file=etc/karaf/java.util.logging.properties
karaf.data=../sonatype-work/nexus3
java.io.tmpdir=../sonatype-work/nexus3/tmp
karaf.startLocalConsole=false
i4j.vpt=true
sun.java.command=com.install4j.runtime.launcher.UnixLauncher start 9d17dc87   org.sonatype.nexus.karaf.NexusMain
sun.java.launcher=SUN_STANDARD
</properties>
</vm_arguments>
<tty>
<blob name='MethodHandlesAdapterBlob' size='32000'>
<sect index='1' size='32000' free='31628'/>
</blob>
<writer thread='8451'/>
<dependency_failed type='abstract_with_unique_concrete_subtype' ctxk='java/nio/charset/CharsetEncoder' x='sun/nio/cs/UTF_8$Encoder' witness='sun/nio/cs/ISO_8859_1$Encoder' stamp='1.231'/>
<dependency_failed type='unique_concrete_method' ctxk='sun/nio/cs/ArrayEncoder' x='sun/nio/cs/UTF_8$Encoder encode ([CII[B)I' witness='sun/nio/cs/ArrayEncoder' stamp='1.231'/>
<dependency_failed type='unique_concrete_method' ctxk='java/lang/ThreadLocal' x='java/lang/ThreadLocal getMap (Ljava/lang/Thread;)Ljava/lang/ThreadLocal$ThreadLocalMap;' witness='java/lang/InheritableThreadLocal' stamp='1.420'/>
<dependency_failed type='unique_concrete_method' ctxk='java/lang/ThreadLocal' x='java/lang/ThreadLocal getMap (Ljava/lang/Thread;)Ljava/lang/ThreadLocal$ThreadLocalMap;' witness='java/lang/InheritableThreadLocal' stamp='1.420'/>
<dependency_failed type='unique_concrete_method' ctxk='java/lang/ThreadLocal' x='java/lang/ThreadLocal getMap (Ljava/lang/Thread;)Ljava/lang/ThreadLocal$ThreadLocalMap;' witness='java/lang/InheritableThreadLocal' stamp='1.420'/>
<dependency_failed type='leaf_type' ctxk='java/util/Properties' witness='java/security/Provider' stamp='1.422'/>
<dependency_failed type='leaf_type' ctxk='java/util/Properties' witness='java/security/Provider' stamp='1.422'/>
<dependency_failed type='leaf_type' ctxk='org/apache/felix/framework/wiring/BundleCapabilityImpl' witness='org/apache/felix/framework/ServiceRegistrationImpl$ServiceReferenceImpl' stamp='1.502'/>
<dependency_failed type='leaf_type' ctxk='org/apache/felix/framework/wiring/BundleCapabilityImpl' witness='org/apache/felix/framework/ServiceRegistrationImpl$ServiceReferenceImpl' stamp='1.502'/>
<dependency_failed type='unique_concrete_method' ctxk='java/net/URLStreamHandler' x='java/net/URLStreamHandler setURL (Ljava/net/URL;Ljava/lang/String;Ljava/lang/String;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V' witness='org/apache/felix/framework/URLHandlersStreamHandlerProxy' stamp='1.511'/>
<dependency_failed type='unique_concrete_method' ctxk='java/net/URLStreamHandler' x='java/net/URLStreamHandler equals (Ljava/net/URL;Ljava/net/URL;)Z' witness='org/apache/felix/framework/URLHandlersStreamHandlerProxy' stamp='1.511'/>
<dependency_failed type='leaf_type' ctxk='java/security/cert/Certificate' witness='java/security/cert/X509Certificate' stamp='1.825'/>
<writer thread='24323'/>
<dependency_failed type='unique_concrete_method' ctxk='java/lang/Throwable' x='java/lang/Throwable fillInStackTrace ()Ljava/lang/Throwable;' witness='com/sun/org/apache/xerces/internal/impl/XMLEntityScanner$1' stamp='3.013'/>
<dependency_failed type='unique_concrete_method' ctxk='java/lang/Throwable' x='java/lang/Throwable fillInStackTrace ()Ljava/lang/Throwable;' witness='com/sun/org/apache/xerces/internal/impl/XMLEntityScanner$1' stamp='3.013'/>
<dependency_failed type='unique_concrete_method' ctxk='java/lang/Throwable' x='java/lang/Throwable fillInStackTrace ()Ljava/lang/Throwable;' witness='com/sun/org/apache/xerces/internal/impl/XMLEntityScanner$1' stamp='3.013'/>
<dependency_failed type='unique_concrete_method' ctxk='java/lang/Throwable' x='java/lang/Throwable fillInStackTrace ()Ljava/lang/Throwable;' witness='com/sun/org/apache/xerces/inter

[eyalkoren]

not setting the boot_delegation_packages at all causes the problem to apear again.

This is expected. However,

leave the option boot_delegation_packages empty and dont append co.elastic.apm.agent.* to \karaf\etc\config.properties fixed also the problem

means that the OSGi allowed delegating loading of our classes, which is confusing…

In any case, I am glad this provides a solution. If you get an error of our classes not found, just use one of the options 1/2.

Thanks for testing and reporting!

[gruselglatz]

@eyalkoren I’ve tried it now, this is the result:

Option 1 -> Fixed the problem Option 2 -> Fixed the problem

Option 2a: leave the option boot_delegation_packages empty and dont append co.elastic.apm.agent.* to \karaf\etc\config.properties fixed also the problem.

But not setting the boot_delegation_packages at all causes the problem to apear again.

So for me it’s fixed with the new option, regardless of the setting of it.

Thank you very much! 👍