Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Enabling build statistics results in extremely slow response ( > 15 min or hanging forever) from the /prometheus endpoint

See original GitHub issue

Steps to reproduce

Enable build statistics by checking the corresponding checkboxes

Expected behaviour

The /prometheus endpoint should start returning build statistics

Actual behaviour

The response hangs for more than 15 min or forever. Because Prometheus server keeps sending requests, we can observe dozens of waiting threads like the one below

Handling GET /prometheus/ from 10.134.100.1 : qtp1535128843-372
"Handling GET /prometheus/ from 10.134.100.1 : qtp1535128843-372" Id=372 Group=main WAITING on com.google.common.util.concurrent.AbstractFuture$Sync@e8004d8
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on com.google.common.util.concurrent.AbstractFuture$Sync@e8004d8
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:132)
	at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3586)
	at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2333)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2222)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.getNode(SimpleXStreamFlowNodeStorage.java:101)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.getNode(CpsFlowExecution.java:1793)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:1181)
	at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:165)
	at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:156)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.getSortedStageNodes(FlowNodes.java:72)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.getSortedStageNodes(FlowNodes.java:68)
	at org.jenkinsci.plugins.prometheus.JobCollector.appendJobMetrics(JobCollector.java:269)
	at org.jenkinsci.plugins.prometheus.JobCollector$1.invoke(JobCollector.java:155)
	at org.jenkinsci.plugins.prometheus.JobCollector$1.invoke(JobCollector.java:136)
	at org.jenkinsci.plugins.prometheus.util.Jobs.forEachJob(Jobs.java:21)
	at org.jenkinsci.plugins.prometheus.JobCollector.collect(JobCollector.java:136)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:180)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:144)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:165)
	at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:127)
	at org.jenkinsci.plugins.prometheus.MetricsRequest$1.generateResponse(MetricsRequest.java:22)
	at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
	at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
	at org.kohsuke.stapler.Function.renderResponse(Function.java:164)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:147)
	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:482)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:458)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
	at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:503)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)

Eventually, Jenkins would become irresponsive and would require restart

Server configuration

Operating system: We are running Jenkins in Kubernetes in GCP in a Linux container; the image https://hub.docker.com/r/jenkinsci/jenkins , tag: 2.164.2 Jenkins Version: 2.164.2 Plugin Version: 2.0.0

Issue Analytics

State:
Created 4 years ago
Reactions:5
Comments:6 (1 by maintainers)

Top GitHub Comments

1reaction

wojciechzachwiejacommented, Aug 23, 2019

I encountered similar issue. This change https://github.com/jenkinsci/prometheus-plugin/commit/a1a5614a77394e66a829da53d4d326c4ef83ea55 fixed slow response in my case.

0reactions

sestocommented, Jul 26, 2019

The config is pretty much standard:

- job_name: tools/jenkins-monitor/0
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - tools
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_jenkins_monitor]
    separator: ;
    regex: .+
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: ui
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: ui
    action: replace

Top Results From Across the Web

Getting started - Prometheus.io

This guide is a "Hello World"-style tutorial which shows how to install, configure, and use a simple Prometheus instance. You will download and...

Troubleshooting Omnibus GitLab installation issues

On SELinux-enabled systems the Git user's .ssh directory or its contents can get their security context messed up. You can fix this by...

All configuration options - Quarkus

AWS Lambda Type Default AWS Lambda Common Type Default AWS Lambda Gateway REST API Type Default Agroal ‑ Database connection pool Type Default

How to troubleshoot remote write issues in Prometheus

A breakdown of all the metrics and configurations that can help you diagnose a remote write issue in Prometheus.

VictoriaMetrics · The High Performance Open Source Time ...

VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. VictoriaMetrics is available i...