question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Prometheus] Notify the user when trying to report duplicate metrics

See original GitHub issue

Some libraries happen to report the same metric twice. Although this use case is not clear (to me, see https://github.com/spring-projects/spring-boot/issues/6404) - it happens.

As for spring-boot POV this is OK , it’s clearly not a valid report from Prometheus perspective.

What I suggest is to prevent it while gathering the metrics before the report, notify the user of the error, and prevent the continuation of the report (probably by an exception)

Code reference:
Verify uniqueness of MetricFamilySamples.Sample here: https://github.com/micrometer-metrics/micrometer/blob/5fbaa762afd71b5091e16e165f008ba1e1fe0c18/implementations/micrometer-registry-prometheus/src/main/java/io/micrometer/prometheus/MicrometerCollector.java#L88 and throw an exception if this isn’t a valid report (i.e with duplicate metrics)

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
mweirauchcommented, Sep 7, 2018

I can only second Jons statements. How shall we hinder two independant pieces of code from registering the same metric? If these pieces of code do by chance register the same metric, there will never be two metrics registered within the MeterRegistries. (If my naive memory serves me right, that possibility was actually the case in the old Actuator framework, but this isn’t transferable to Micrometer.)

So the proposed sanitization/grouping in the MicrometerCollector has no effect.

The only thing I could think of is hindering registration of an already existing metric, but this will for sure break everything with its current get-or-create-style. (Correct me if I am wrong Jon.)

1reaction
jkschneidercommented, Sep 7, 2018

I guess what I was trying to say was that there is no possibility for duplication at this point in the code. Micrometer has already guaranteed name+tag uniqueness at the registry level.

Accidental usage of the same set of name and tags for completely separate purposes in your codebase is not a condition we can detect programmatically. It’s a semantic concern.

Read more comments on GitHub >

github_iconTop Results From Across the Web

De-duplication of Prometheus alerts for Incidents - GitLab
Users receive alerts through multiple channels and want to check if the GitLab issue pertains to the same group as an alert received...
Read more >
Configuration - Prometheus.io
To specify which configuration file to load, use the --config.file flag. ... This is unlikely to be what you want, unless you have...
Read more >
Collection of alerting rules - Awesome Prometheus alerts
Prometheus has many samples rejected due to duplicate timestamps but different values [copy]. - alert: PrometheusTargetScrapeDuplicate expr: ...
Read more >
Labels in Prometheus alerts: think twice before using them
This receiver config says we want to get notification with common summary ... is uniquely identified by its metric name and a set...
Read more >
Alert manager triggers web-hook repeatedly for same alert
https://prometheus.io/docs/alerting/latest/configuration/ ... this way is that you want to keep getting notification about an active issue, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found