Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Getting errors JMX scrape failed: Failed to retrieve RMIServer stub

See original GitHub issue

Hello,

since I deployed jmx_exporter into our production setup, I’m getting those error on every JMX scrape. It’s quite annoying, as the logs are full of these exceptions and it’s quite hard to search for anything “normal” in the logs now 😃

Any suggestion is welcome Thanks, Miroslav

Edit: surprisingly, the scraping works and we have the numbers in Prometheus/Grafana

Our setup: a Payara (Glassfish) application running inside of a docker container

Docker port mapping: -p 9204:8089

Agent JVM parameter: -javaagent/opt/java-agents/jmx_prometheus_javaagent-0.9.jar=8089\\:/opt/java-agents/jmx_exporter.yaml

YAML config:

---
username: 
password: 

rules:
  - pattern: ".*"

Exception: [#|2017-08-07T14:40:22.560+0000|SEVERE|Payara 4.1|io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector|_ThreadID=10;_ThreadName=jmx_exporter-10;_TimeMillis=1502116822560;_LevelValue=1000;|JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:106) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:401) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:143) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:158) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:128) at java.util.Collections.list(Collections.java:5240) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.MetricsServlet.doGet(MetricsServlet.java:40) at io.prometheus.jmx.shaded.javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at io.prometheus.jmx.shaded.javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.Server.handle(Server.java:365) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) at io.prometheus.jmx.shaded.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at io.prometheus.jmx.shaded.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at io.prometheus.jmx.shaded.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at io.prometheus.jmx.shaded.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) at javax.naming.InitialContext.lookup(InitialContext.java:417) at javax.naming.InitialContext.lookup(InitialContext.java:417) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ... 30 more Caused by: java.rmi.ConnectIOException: non-JRMP server at remote endpoint at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:248) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118) ... 36 more