question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

java agent reporting high cardinality metrics for http_client_duration_bucket

See original GitHub issue

Hi, I am using otel java agent 1.10.1 And I see metrics like below generated for my http traffic:

http_client_duration_bucket{http_flavor="1.1",http_method="GET",http_status_code="200",http_url="https://idp.vault.dev-local.prisidio.net/.well-known/jwks.json",le="10"} 1
http_client_duration_bucket{http_flavor="1.1",http_method="GET",http_status_code="500",http_url="http://localhost:9900/stux/v1/users/92108123607311744",le="10"} 1

The http_url is logging full URL path, which produces high cardinality metrics. Looking at specs, I thought http.url should be using variable for path parameter to prevent high cardinality metrics. Is this the default expected behaviour? Is it possible to disable http.client.duration metric? or disabling default http metrics altogether? Any guidance is much appreciated. I couldn’t find much resources/discussions regarding this.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

3reactions
anuraagacommented, Feb 4, 2022

Hi @gautam-nutalapati - we have been looking at cardinality of server metrics but may have forgotten about client. I agree that we shouldn’t be populating the URL there. @mateuszrzeszutek does that sound right?

1reaction
mateuszrzeszutekcommented, Feb 7, 2022

Is there a way to disable these HTTP client metrics meanwhile when using auto-instrumentation java agent?

No, unfortunately there is no simple way to do that. Perhaps it could be achieved with implementing an extension and registering a custom view, e.g.

meterProviderBuilder
  .registerView(
    InstrumentSelector.builder()
      .setInstrumentName("http.client.duration")
      .build(),
    View.builder()
      .filterAttributes(key -> !key.equals("http.url"))
      .build())

I agree that we shouldn’t be populating the URL there. @mateuszrzeszutek does that sound right?

Yes, we should definitely fix that. Currently we’re already trimming the query from the http.url attribute; should we remove the path too? Or just include the net.* attributes instead of the URL.

CC @jsuereth

Read more comments on GitHub >

github_iconTop Results From Across the Web

Ingest high-cardinality logs to generate metrics with ...
Explains how to use CloudWatch embedded metric format to ingest high-cardinality logs and extract metrics from them.
Read more >
Understand and query high cardinality metrics
For New Relic Query Language (NRQL): how to understand high cardinality metrics and successfully query for them using NRQL.
Read more >
aws-embedded-metrics-java/README.md at master - GitHub
This is useful for contextual and potentially high-cardinality data that is not appropriate for CloudWatch Metrics dimensions. Requirements: Length 1-255 ...
Read more >
Scaling up reporting on high-cardinality metrics - Segment
Scaling up reporting on high-cardinality metrics ... Unfortunately, we can't do this for calculating cardinality since the sum of the number ...
Read more >
Monitoring High Cardinality Metrics | MetricFire Blog
Hosted Graphite keeps all the benefits of open-source Graphite and further enhances the tool with the built-in agent, team accounts, granular ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found