java.lang.OutOfMemoryError: Java heap space

Steps to reproduce

Install Plugin … the issue might be having loads of jenkins jobs …

Expected behaviour

No SEVERE h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (prometheus_async_worker thread/37408) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code. is logged.

Actual behaviour

Seeing this appearing in logs

 [id=37408]	SEVERE	h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (prometheus_async_worker thread/37408) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.

Server configuration

                "JAVA_OPTS=-Dgroovy.use.classvalue=true -Djava.awt.headless=true -Dhudson.matrix.MatrixConfiguration.useShortWorkspaceName=true -Dhudson.model.ParametersAction.keepUndefinedParameters=true -Dhudson.model.DirectoryBrowserSupport.CSP=",
                "PATH=/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "JAVA_HOME=/usr/local/openjdk-8",
                "JAVA_VERSION=8u242",
                "JAVA_BASE_URL=https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u242-b08/OpenJDK8U-jdk_",
                "JAVA_URL_VERSION=8u242b08",
                "JENKINS_HOME=/var/jenkins_home",
                "JENKINS_SLAVE_AGENT_PORT=50000",
                "REF=/usr/share/jenkins/ref",
                "JENKINS_VERSION=2.263.1",
                "JENKINS_UC=https://updates.jenkins.io",
                "JENKINS_UC_EXPERIMENTAL=https://updates.jenkins.io/experimental",
                "JENKINS_INCREMENTALS_REPO_MIRROR=https://repo.jenkins-ci.org/incrementals",
                "COPY_REFERENCE_FILE_LOG=/var/jenkins_home/copy_reference_file.log",
                "TZ=America/Los_Angeles"

Operating system:

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Jenkins Version: Jenkins docker image jenkins/jenkins:lts

Plugin Version:

2.0.8

Notify

@markyjackson-taulia

[kyounger]

What might be nice is to split the creation of those metrics for pipelines into an additional plugin. If a user needed them, then they could optionally include it. Feels like this plugin should keep it simple and stick to just converting the already-defined metrics into what prometheus needs. I do think the pipeline metrics can be useful, and perhaps reworking them is necessary, but would hate to see them disappear entirely.

[Vlatombe]

This is very likely to be caused by https://github.com/jenkinsci/prometheus-plugin/pull/177

This plugin currently generates metrics per job/build/stage and this is a very bad practice.

• Every new job adds 12 metrics.
• Every new stage adds 2 metrics
• Every new build adds 4 metrics.

https://prometheus.io/docs/practices/naming/

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

The memory it takes to store all these metrics explodes with the number of jobs, stages and builds in the Jenkins instance, and the corresponding storage on the Prometheus side will also explode.