question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Prometheus plugins exception in log results in empty page when scraping

See original GitHub issue

Hello,

Steps to reproduce

Use prometheus-plugin: 2.0.8 and Jenkins: 2.235.3 with a large number of jobs and scrape them, an empty scraping page will be returned

Expected behaviour Show the scraped output on the /prometheus endpoint

Actual behaviour When navigating to the webpage/prometheus, I am receiving an empty page with a 200 status code. In my logs I see:

A thread (prometheus_async_worker thread/767) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.NullPointerException
	at org.jenkinsci.plugins.prometheus.JobCollector.lambda$appendJobMetrics$1(JobCollector.java:310)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at org.jenkinsci.plugins.prometheus.JobCollector.appendJobMetrics(JobCollector.java:310)
	at org.jenkinsci.plugins.prometheus.JobCollector.lambda$collect$0(JobCollector.java:220)
	at org.jenkinsci.plugins.prometheus.util.Jobs.forEachJob(Jobs.java:14)
	at org.jenkinsci.plugins.prometheus.JobCollector.collect(JobCollector.java:214)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144)
	at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
	at org.jenkinsci.plugins.prometheus.service.DefaultPrometheusMetrics.collectMetrics(DefaultPrometheusMetrics.java:51)
	at org.jenkinsci.plugins.prometheus.service.PrometheusAsyncWorker.execute(PrometheusAsyncWorker.java:41)
	at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
	at java.lang.Thread.run(Thread.java:748)

I am also seeing

A thread (prometheus_async_worker thread/1054) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.NullPointerException

In the prometheus logs, it seems to still receive information correctly from jobs, but I got no output when jenkins is scraped.

Server configuration prometheus-plugin: 2.0.8 Jenkins: 2.235.3

Settings for the plugins is the default settings.

Notify @markyjackson-taulia

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:2
  • Comments:6

github_iconTop GitHub Comments

2reactions
olafrauchcommented, Nov 3, 2020

Similar issue here. Since plugin 2.0.8 upgrade: There is a ClassCastException in the log and the /prometheus page is empty

SEVERE h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (prometheus_async_worker thread/3301139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code. java.lang.ClassCastException

Note: As soon as I disable the five “Count Duration *” settings of the plugin in “Manage Jenkins”, the error log entry is gone and the (now less) metrics are back.

0reactions
github-actions[bot]commented, Jan 3, 2021

Stale issue message

Read more comments on GitHub >

github_iconTop Results From Across the Web

Prometheus API return blank page · Issue #82 - GitHub
As I try to setup the log with org.jenkinsci.plugins.prometheus. Any thing prints out, so I guess the prometheus plugin doesn't execute.
Read more >
Writing exporters - Prometheus.io
The one exception is when you're exporting the same data with different labels via multiple metrics, in which case that's usually the sanest...
Read more >
4. Exposition - Prometheus: Up & Running [Book] - O'Reilly
Each time Prometheus went to scrape the application, it would randomly get the metrics from only one of the workers, which would be...
Read more >
Monitor your logging pipeline with Prometheus Operator
Monitor your logging pipeline with Prometheus Operator. You can configure the Logging operator to expose metrics endpoints for FluentD and Fluent-bit using ...
Read more >
How to export metrics (collected by DropWizard) in ...
I have implemented different ways to export metrics to Prometheus: 1.1) Custom implementation for Pushgateway.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found