Stack overflow when scraping metrics
See original GitHub issueHello,
I’m trying to scrape the prometheus metrics endpoint however always get a timeout when using a proxy. It only occurred to me today to try and poke the endpoint directly with a generous timeout for curl and here’s what I got:
java.lang.StackOverflowError
at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:149)
at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:137)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
(...)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
Caused: javax.servlet.ServletException
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:765)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
at org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:51)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run
I’m wondering if this could be related to #10? I do have dozens of folders (most of them multibranch pipelines) and jobs and thousands of builds (if not more) which I imagine doesn’t help!
edit: any subsequent GETs to /prometheus
result in an empty 200 OK.
Issue Analytics
- State:
- Created 6 years ago
- Comments:16 (5 by maintainers)
Top Results From Across the Web
Can I just parse my log files and expose it to a http API for ...
Yes. As long as the scrape target is formatted per Prometheus exposition format, you can generate the metrics however you wish.
Read more >Migrating metrics from InfluxDB to M3 - Stack Overflow Blog
Once the data is in Kafka, we then distribute it to various metrics backends, internal or external, again using Telegraf. The final piece...
Read more >Scraping different Prometheus metrics at different rates
1 Answer 1 ... You can define a different scrape_interval in the scrape config of two different jobs: a job for slow polling...
Read more >Reducing Prometheus metrics usage with relabeling - Grafana
To learn more about regex match groups, please see this StackOverflow answer. ... and endpoints you want to scrape (or drop) to tune...
Read more >Do prometheus metrics have some sort of "freshness" lifetime?
I'm scraping some metrics (openstack cinder volume sizes) every 15 minutes, and the results produce a discontinuous graph, like this:.
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@moertel I’ve not tried this for a while but from what I remember I’ve restarted Jenkins and used curl with huge timeout values directly against the Jenkins http port (avoiding load balancers and proxies to avoid a 504 interrupting the connection). If the connection gets interrupted or you get a stack overflow, every subsequent request to
/prometheus
will return an empty body with 200 OK.@markyjackson-taulia thanks for looking into the issue. Sorry to read you weren’t able to reproduce it. From what I gather, there have to be quite some jobs in Jenkins (for me: 139 jobs and approx. 30 builds each) to cause this resource exhaustion. Having said that, while this makes producing a minimal example rather difficult, the issue is still real.
So are you closing the issue because you categorise it as an edge case not worth fixing? Or because you would want to fix it but there’s not enough information for you to start troubleshooting?
Either way, I’d be happy to assist; let me know what you need from me.