Won't run on machine with more than 20 CPU cores

I’m using the JMX Exporter to monitor Apache Spark, using it as a Java agent. This works fine until I try and run it on a server with more than 20 cores. In the spark logs I see the following line as spark is starting up:

2017-03-15 21:35:50.606:WARN:ipjsoejs.AbstractConnector:insufficient threads configured for SelectChannelConnector@0.0.0.0:19105

Then once spark is running the log gets spammed with lines like the following:

2017-03-15 21:35:53.858:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@3712cd1e{l(/192.168.92.52:44556)<->r(/192.168.76.1:19105),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0}-{AsyncHttpConnection@905c2db,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@5f9d5ea6 2017-03-15 21:35:53.859:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@3712cd1e{l(/192.168.92.52:44556)<->r(/192.168.76.1:19105),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@905c2db,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@5f9d5ea6 2017-03-15 21:35:53.859:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@3712cd1e{l(/192.168.92.52:44556)<->r(/192.168.76.1:19105),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@905c2db,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@5f9d5ea6

Not only is the exporter not working in this situation but spark isn’t running correctly either and my disk is filling up very quickly!

To reproduce this is simple. Use a machine with >20 cores, add the JMX exporter as a java agent to any java process and watch the logs. You can prove the 20 cores thing by disabling cores until you have <=20 by doing the following (on linux):

echo 0 > /sys/devices/system/cpu/cpuX/online

[ihulsbus]

Hi, We are also experiencing this issue at random moments with javaagent 0.9 and 10 machine cores.

Am I correct to say that the switch to HTTPserv in version 0.10 should be a fix for this issue?

[bencoughlan]

Upgrade to the 1.0 engine, it’ll sort everything

Sent with BlackBerry Work (www.blackberry.com)

---

From: anton-opsguru [e-mail] Sent: 27 Sep 2017 20:37 To: prometheus/jmx_exporter [e-mail] Cc: Ben Coughlan [e-mail]; Comment [e-mail] Subject: Re: [prometheus/jmx_exporter] Won’t run on machine with more than 20 CPU cores

This issue is still easily reproducible on 20+ core machines on . Hitting: java.lang.IllegalStateException: Insufficient threads: max=10 < needed(acceptors=3 + selectors=8 + request=1) I am running on :

• Java 1.8_144
• jmx_prometheus_javaagent-0.10.jar Full stack trace:

2017-09-27 18:49:23.541:INFO::main: Logging initialized @7596ms to io.prometheus.jmx.shaded.org.eclipse.jetty.util.log.StdErrLog 2017-09-27 18:49:26.138:INFO:ipjsoejs.Server:main: jetty-9.4.z-SNAPSHOT Exception in thread “main” java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) Caused by: java.lang.IllegalStateException: Insufficient threads: max=10 < needed(acceptors=3 + selectors=8 + request=1) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.Server.doStart(Server.java:414) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:84) … 6 more FATAL ERROR in native method: processing of -javaagent failed

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/prometheus/jmx_exporter/issues/122#issuecomment-332632515, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AY3BAaoy4pKPq1Q-k6HGZ4BGqjIaade5ks5smqQEgaJpZM4MemZR.