http_server_duration_bucket generates high cardinality metrics

Describe the bug The Java agent, when configured to export metrics with Prometheus exporter, creates as expected the bucket http_server_duration_bucket.

This metric also has the http_route label, which produces high cardinality because it contains full paths like:

http_server_duration_bucket{container="xxx", endpoint="xxx", http_flavor="1.1", http_host="xxx", http_method="GET", http_route="/app/wsrest/userlogo/holding.png", http_scheme="http", http_status_code="200", instance="xxx:9464", job="xxx", le="7500.0", namespace="wms", pod="xxxx", service="wms"}

In my cluster this caused the number of time series to increase of ~300k !

Steps to reproduce

1. Instrument a Java app to export metrics with Prometheus exporter
2. Setup a Prometheus instance to scrape those metrics
3. Query Prometheus for http_server_duration_bucket

What did you expect to see? http_server_duration_bucket should export lower cardinality metrics

What did you see instead? http_server_duration_bucket export high cardinality metrics

What version are you using? 1.12.1

Environment Compiler: (e.g., “AdoptOpenJDK 11.0.6”) OS: Ubuntu 20.04 Runtime (if different from JDK above): (e.g., “Oracle JRE 8u251”) OS (if different from OS compiled on): (e.g., “Windows Server 2019”)

Additional context Add any other context about the problem here.

[irizzant]

Hi @mateuszrzeszutek @trask

Would it be possible for you to extract a part of your application as a repro app?

this request took me a lot of work!

Anyway, please check https://github.com/irizzant/otel-java-instrumentation-5753

If you run the reproducer, you will see that the exported Undertow metrics have http.route that get to the very single resources, for example you can see http.route reporting the single /jsf-demo/hello.xhtml

[mateuszrzeszutek]

Hey @irizzant , Would it be possible for you to extract a part of your application as a repro app? Without a working example it’s really hard to reason about the http.route attribute, since it usually isn’t set by the server instrumentations, but (one or more) controller/mvc framework.