Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add option to stop reporting deleted resources in Prometheus scrape endpoint

See original GitHub issue

Proposal

Add a setting to stop reporting deleted resources in Prometheus scrape endpoint.

From https://docs.promitor.io/faq#how-does-promitor-handle-deleted-resources

Removed resources will immediately stop being scraped by Promitor, but still be reported in Prometheus scrape endpoint.

Reporting metrics for deleted resources:

can be misleading to those viewing the metrics (e.g. this is not standard behaviour for most exporters)
leads to false-positive alerts with promitor_scrape_error metric, as the metric returns 1 for any deleted resources
unnecessarily stores additional time-series in Prometheus

An option to stop reporting metrics for deleted resources would resolve the above issues.

Component

Scraper

Contact Details

No response

Issue Analytics

State:
Created 2 years ago
Comments:7 (6 by maintainers)

Top GitHub Comments

1reaction

tomkerkhovecommented, Jan 24, 2022

This is coming as per https://github.com/prom-client-net/prom-client/discussions/177 and https://github.com/prom-client-net/prom-client/issues/178 so then we can do a more proper clean-up.

However, we would need to check if this is an opt-in or not given others who use timestamps might want to keep them around.

0reactions

tomkerkhovecommented, Sep 12, 2022

Yup

Top Results From Across the Web

Configuration | Prometheus

A scrape_config section specifies a set of targets and parameters describing how to scrape them. In the general case, one scrape configuration specifies...

prometheus.scrape | Grafana Agent documentation

prometheus.scrape reports the status of the last scrape for each configured scrape job on the component's debug endpoint. Debug metrics.

Scraping additional Prometheus sources and importing those ...

Tutorial for adding a new Prometheus scrape target: Prometheus API Server metrics. The Kubernetes API Server exposes Prometheus metrics on endpoints by default....

Prometheus Operator

To remove the operator and Prometheus, first delete any custom resources you created in each namespace. The operator will automatically shut down and...

Kubernetes Monitoring with Prometheus, Ultimate Guide

Prometheus monitoring is fast becoming the Docker and Kubernetes monitoring tool to use. Learn to implement Kubernetes monitoring with Prometheus.