Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ResultProcessor NPEs after running for a while

See original GitHub issue

After running perfectly fine for a while, jmxtrans stops writing metrics to the output writer. I see this error in the logs multiple times (I’m assuming for each attempted query):

[06 Nov 2016 23:16:08] [jmxtrans-result-5] 197814012 WARN  (com.googlecode.jmxtrans.jmx.ResultProcessor$1:60) - Could not write results [Result(attributeName=OneMinuteRate, className=com.yammer.metrics.reporting.JmxReporter$Meter, objDomain=kafka.server, typeName=type=ReplicaManager,name=IsrShrinksPerSec, values={OneMinuteRate=0.0}, epoch=1478474167305, keyAlias=replication.isr.shrink)] of query Query(objectName=kafka.server:type=ReplicaManager,name=IsrShrinksPerSec, keys=[], attr=[OneMinuteRate], typeNames=[], resultAlias=replication.isr.shrink, useObjDomainAsKey=false, allowDottedKeys=false, useAllTypeNames=false, outputWriterInstances=[]) to output writer com.googlecode.jmxtrans.model.output.support.ResultTransformerOutputWriter@2505453a
java.lang.NullPointerException
        at com.googlecode.jmxtrans.model.output.support.WriterPoolOutputWriter.doWrite(WriterPoolOutputWriter.java:59)
        at com.googlecode.jmxtrans.model.output.support.ResultTransformerOutputWriter.doWrite(ResultTransformerOutputWriter.java:50)
        at com.googlecode.jmxtrans.jmx.ResultProcessor$1.run(ResultProcessor.java:58)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

A few lines before the first such occurrence, I found this error:

[06 Nov 2016 23:16:07] [jmxtrans-result-0] 197813012 ERROR (com.googlecode.jmxtrans.model.output.support.pool.WriterPoolable:57) - Could not flush writer
java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flush(BufferedWriter.java:254)
        at com.googlecode.jmxtrans.model.output.support.pool.TimeBasedFlush.flush(TimeBasedFlush.java:51)
        at com.googlecode.jmxtrans.model.output.support.pool.WriterPoolable.release(WriterPoolable.java:54)
        at com.googlecode.jmxtrans.model.output.support.WriterPoolOutputWriter.doWrite(WriterPoolOutputWriter.java:59)
        at com.googlecode.jmxtrans.model.output.support.ResultTransformerOutputWriter.doWrite(ResultTransformerOutputWriter.java:50)
        at com.googlecode.jmxtrans.jmx.ResultProcessor$1.run(ResultProcessor.java:58)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

So I’m assuming that this is caused by some form of network failure, and then the connections are not cleared properly or something like that. I tried digging into the code, but there are a bit too many layers of abstraction for me to figure something out quickly.

I’m using jmxtrans-259 with the GraphiteOutputWriter - let me know if you need any other details. Thanks in advance!