Telegraph error reading prometheus-plugin output.

Hi,

We are seeing the following error reported in Telegraf when trying to read the output of the jenkinsci/prometheus-plugin.

telegraf[7915]: 2021-04-20T13:52:30Z E! [inputs.prometheus] Error in plugin: error reading metrics for http://<host>/prometheus/: reading text format failed: text format parsing error in line 1394: second TYPE line for metric name “jenkins_node_builds”, or TYPE reported after samples

Yesterday it seemed this error occured on the jenkins instance which had the recently released version prometheus:2.0.10 installed but I subsequently upgraded another (test) instance to the same version prometheus:2.0.10 and the problem was not reported…

I checked the outputs of the prometheus plugin at <jenkins-url>/<prometheus-metric-endpoint> and I found on the working version it was not outputting a “jenkins_node_builds” metric section at all. On the version that is failing it outputs multiple “jenkins_node_builds” sections but the first section is missing the #HELP and #TYPE lines. This fits with Telegraf’s report as it first sees a metric section with no #HELP or #TYPE for “jenkins_node_builds” and then later on finds the #HELP and # TYPE lines for the next section so in it’s eyes the header lines are coming after the section they should be attached to?? (I’m only guessing here)

I’ve included the Version report for both the working version and the broken version as I don’t know where the “jenkins_node_builds” section is originating from and it could be the plugin outputting that metric might be the problem

Anyway hope this helps Cheers Phill

Version report

Broken Version

Result
Jenkins: 2.249.1
OS: Linux - 4.15.0-1113-azure
---
blueocean-pipeline-editor:1.23.2
github-branch-source:2.9.0
resource-disposer:0.15
blueocean-rest-impl:1.23.2
scm-api:2.6.4
blueocean-pipeline-scm-api:1.23.2
docker-workflow:1.24
jdk-tool:1.0
command-launcher:1.2
variant:1.3
blueocean-dashboard:1.23.2
workflow-step-api:2.23
blueocean-personalization:1.23.2
git-client:3.4.2
token-macro:2.13
blueocean-git-pipeline:1.23.2
pipeline-stage-step:2.5
blueocean-core-js:1.23.2
structs:1.22
htmlpublisher:1.23
mercurial:2.10
junit:1.49
jquery:1.12.4-1
bouncycastle-api:2.16.0
thinBackup:1.10
ssh-credentials:1.18.1
analysis-model-api:9.8.0
build-monitor-plugin:1.12+build.201809061734
matrix-auth:2.6.3
workflow-basic-steps:2.21
favorite:2.3.2
workflow-durable-task-step:2.36
snakeyaml-api:1.27.0
github:1.31.0
pipeline-utility-steps:2.7.1
blueocean-bitbucket-pipeline:1.23.2
workflow-cps:2.90
font-awesome-api:5.15.2-1
active-directory:2.23
blueocean-jwt:1.23.2
blueocean-web:1.23.2
docker-java-api:3.1.5.2
influxdb:3.0
ws-cleanup:0.39
checks-api:1.5.0
jenkins-design-language:1.23.2
pipeline-model-definition:1.7.2
bitbucket:1.1.27
credentials-binding:1.23
pipeline-model-api:1.7.2
blueocean-config:1.23.2
plugin-util-api:1.7.1
lockable-resources:2.10
cloudbees-bitbucket-branch-source:2.9.2
handlebars:3.0.8
prometheus:2.0.10
pipeline-rest-api:2.19
jaxb:2.3.0.1
azure-credentials:4.0.6
ssh-agent:1.22
ssh-slaves:1.31.5
jackson2-api:2.12.3
blueocean-i18n:1.23.2
credentials:2.3.18
mailer:1.34
pipeline-build-step:2.13
blueocean:1.23.2
cppcheck:1.25
blueocean-events:1.23.2
matrix-project:1.18
rebuild:1.32
git-server:1.9
build-timeout:1.20
cloud-stats:0.27
cloudbees-folder:6.14
workflow-support:3.8
jira:3.1.3
warnings-ng:8.9.2
blueocean-commons:1.23.2
jsch:0.1.55.2
build-pipeline-plugin:1.5.8
display-url-api:2.3.4
azure-acs:1.0.4
popper-api:1.16.1-1
workflow-multibranch:2.22
azure-commons:1.0.5
metrics:4.0.2.7
antisamy-markup-formatter:2.1
plain-credentials:1.7
conditional-buildstep:1.4.1
sse-gateway:1.23
apache-httpcomponents-client-4-api:4.5.13-1.0
git:4.4.2
blueocean-rest:1.23.2
pipeline-stage-tags-metadata:1.7.2
durable-task:1.35
pipeline-milestone-step:1.3.1
extended-read-permission:3.2
jquery-detached:1.2.1
docker-commons:1.17
forensics-api:0.10.1
jira-steps:1.6.0
kubernetes-cd:2.3.1
pipeline-graph-analysis:1.10
blueocean-autofavorite:1.2.4
github-api:1.116
trilead-api:1.0.13
blueocean-github-pipeline:1.23.2
email-ext:2.82
workflow-api:2.42
azure-app-service:1.0.2
maven-plugin:3.8
momentjs:1.1.1
blueocean-pipeline-api-impl:1.23.2
pipeline-model-extensions:1.7.2
timestamper:1.12
pubsub-light:1.13
copyartifact:1.46
script-security:1.76
parameterized-trigger:2.39
jquery3-api:3.5.1-2
workflow-cps-global-lib:2.17
data-tables-api:1.10.23-2
azure-vm-agents:1.5.1
ace-editor:1.1
ant:1.8
windows-slaves:1.0
external-monitor-job:1.4
ldap:1.11
pam-auth:1.5.1
violations:0.7.11
blueocean-jira:1.23.2
bootstrap4-api:4.6.0-1
okhttp-api:3.14.9
plot:2.1.9
authentication-tokens:1.4
windows-azure-storage:1.1.7
run-condition:1.5
pipeline-stage-view:2.19
workflow-scm-step:2.11
pipeline-input-step:2.12
javadoc:1.6
blueocean-display-url:2.4.0
echarts-api:4.9.0-3
handy-uri-templates-2-api:2.1.8-1.0
branch-api:2.6.2
workflow-job:2.40

Working Version

Result
Jenkins: 2.277.1
OS: Linux - 4.15.0-1108-azure
---
jackson2-api:2.12.3
cloudbees-folder:6.15
configuration-as-code:1.46
pipeline-stage-tags-metadata:1.7.2
ws-cleanup:0.38
resource-disposer:0.14
antisamy-markup-formatter:2.1
pipeline-github-lib:1.0
ant:1.11
mailer:1.32.1
ssh-agent:1.20
git-client:3.5.1
email-ext:2.79
workflow-cps:2.86
workflow-api:2.42
display-url-api:2.3.4
variant:1.3
workflow-cps-global-lib:2.17
github-api:1.116
credentials-binding:1.24
echarts-api:4.9.0-2
pipeline-input-step:2.12
extended-read-permission:3.2
jquery3-api:3.5.1-2
prometheus:2.0.10
github:1.32.0
workflow-durable-task-step:2.37
workflow-basic-steps:2.22
junit:1.44
snakeyaml-api:1.27.0
pipeline-stage-step:2.5
ace-editor:1.1
pipeline-model-definition:1.7.2
matrix-auth:2.6.5
command-launcher:1.5
structs:1.22
timestamper:1.11.8
trilead-api:1.0.12
git:4.4.5
pipeline-model-api:1.7.2
pipeline-stage-view:2.19
jdk-tool:1.4
pam-auth:1.6
bootstrap4-api:4.5.3-1
jacoco:3.1.0
azure-keyvault:2.1
workflow-aggregator:2.6
token-macro:2.12
ssh-credentials:1.18.1
workflow-support:3.8
bouncycastle-api:2.18
cloudbees-disk-usage-simple:0.10
workflow-step-api:2.23
lockable-resources:2.10
credentials:2.3.14
handlebars:1.1.1
script-security:1.76
pipeline-rest-api:2.19
momentjs:1.1.1
scm-api:2.6.4
workflow-scm-step:2.11
popper-api:1.16.0-7
durable-task:1.35
workflow-job:2.40
pipeline-milestone-step:1.3.1
azure-credentials:4.0.3
cobertura:1.16
workflow-multibranch:2.22
okhttp-api:3.14.9
matrix-project:1.18
gradle:1.36
plugin-util-api:1.4.0
code-coverage-api:1.2.0
git-server:1.9
branch-api:2.6.2
ldap:1.26
ssh-slaves:1.31.2
pipeline-graph-analysis:1.10
jquery-detached:1.2.1
github-branch-source:2.9.1
build-timeout:1.20
metrics:4.0.2.7
plain-credentials:1.7
jsch:0.1.55.2
font-awesome-api:5.15.1-1
checks-api:1.1.1
pipeline-model-extensions:1.7.2
pipeline-build-step:2.13
statistics-gatherer:2.0.3
apache-httpcomponents-client-4-api:4.5.10-2.0

• What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu 16.04.7 LTS (GNU/Linux 4.15.0-1113-azure x86_64)

Reproduction steps

Step 1 - Go to <jenkins-url>/<prometheus-metric-endpoint> Step 2 - Search for ‘jenkins_node_builds’ metric

Results

Expected result:

# HELP jenkins_node_builds Generated from Dropwizard metric import (metric=jenkins.node.builds, type=com.codahale.metrics.Timer)
# TYPE jenkins_node_builds summary
jenkins_node_builds{node="master",quantile="0.5",} 22.496242709
jenkins_node_builds{node="master",quantile="0.75",} 56.998456999000005
jenkins_node_builds{node="master",quantile="0.95",} 56.998456999000005
jenkins_node_builds{node="master",quantile="0.98",} 56.998456999000005
jenkins_node_builds{node="master",quantile="0.99",} 56.998456999000005
jenkins_node_builds{node="master",quantile="0.999",} 56.998456999000005
jenkins_node_builds_count{node="master"} 234.0

Actual result:

# TYPE vm_memory_pools_PS_Survivor_Space_committed_window_1h summary
vm_memory_pools_PS_Survivor_Space_committed_window_1h{quantile="0.5",} 7.2351744E7
vm_memory_pools_PS_Survivor_Space_committed_window_1h{quantile="0.75",} 1.24780544E8
vm_memory_pools_PS_Survivor_Space_committed_window_1h{quantile="0.95",} 1.74063616E8
vm_memory_pools_PS_Survivor_Space_committed_window_1h{quantile="0.98",} 1.91889408E8
vm_memory_pools_PS_Survivor_Space_committed_window_1h{quantile="0.99",} 1.92937984E8
vm_memory_pools_PS_Survivor_Space_committed_window_1h{quantile="0.999",} 1.92937984E8
vm_memory_pools_PS_Survivor_Space_committed_window_1h_count 5388.0
# HELP jenkins_node_docker_agent_11_builds Generated from Dropwizard metric import (metric=jenkins.node.docker-agent-11.builds, type=com.codahale.metrics.Timer)
# TYPE jenkins_node_docker_agent_11_builds summary
jenkins_node_builds{node="docker_agent_11",quantile="0.5",} 0.0
jenkins_node_builds{node="docker_agent_11",quantile="0.75",} 0.0
jenkins_node_builds{node="docker_agent_11",quantile="0.95",} 0.0
jenkins_node_builds{node="docker_agent_11",quantile="0.98",} 0.0
jenkins_node_builds{node="docker_agent_11",quantile="0.99",} 0.0
jenkins_node_builds{node="docker_agent_11",quantile="0.999",} 0.0
jenkins_node_builds_count{node="docker_agent_11"} 0.0
# HELP jenkins_plugins_failed Generated from Dropwizard metric import (metric=jenkins.plugins.failed, type=jenkins.metrics.impl.JenkinsMetricProviderImpl$22)
# TYPE jenkins_plugins_failed gauge
jenkins_plugins_failed 0.0
# HELP vm_count Generated from Dropwizard metric import (metric=vm.count, type=com.codahale.metrics.jvm.ThreadStatesGaugeSet$$Lambda$201/1006609823)
# TYPE vm_count gauge
vm_count 238.0

[Sukiyakijango]

still no fix or workaround for this issue 😦 keeeping us blinde to issue in our jenkins

[Waschndolos]

Correct me if I’m wrong but prometheus:2.0.10 is the latest version right? Which is the version we’re all using.