OpenMetrics collector fails on exemplars

The OpenMetrics collector uses Prometheus’ client lib to parse samples (https://github.com/DataDog/integrations-core/blob/3bf453a/datadog_checks_base/datadog_checks/base/checks/openmetrics/mixins.py#L428). Unlike Prometheus, OpenMetrics has the concept of Exemplars that are causing issues.

Attempts to ingest metrics with exemplars result in:

2021-08-31 14:17:49 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:302 in work) | Error running check openmetrics: [{"message": "Invalid metric name: a_total 1 #", "traceback": "Traceback (most recent call last):
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1006, in run
    self.check(instance)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/base_check.py\", line 135, in check
    self.process(scraper_config)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 533, in process
    for metric in self.scrape_metrics(scraper_config):
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 504, in scrape_metrics
    for metric in self.parse_metric_family(response, scraper_config):
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 428, in parse_metric_family
    for metric in text_fd_to_metric_families(input_gen):
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/libs/prometheus.py\", line 86, in text_fd_to_metric_families
    yield build_metric(sample[0], documentation, typ, [sample])
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/libs/prometheus.py\", line 29, in build_metric
    metric = Metric(name, documentation, typ)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/prometheus_client/metrics_core.py\", line 27, in __init__
    raise ValueError('Invalid metric name: ' + name)
ValueError: Invalid metric name: a_total 1 #

Here you can find the example of a counter with an exemplar that the current collector is unable to ingest: https://github.com/OpenObservability/OpenMetrics/blob/main/tests/testdata/parsers/counter_exemplars/metrics

The proposed solution is to either update the custom text_fd_to_metric_families implementation or use the openmetrics specific parser as suggested by the spec.

[coignetp]

Thank you for the suggestion @mxmCherry , it was indeed missing. Documentation to v2 in #10068

[mxmCherry]

Thanks @coignetp for nice and detailed docs https://github.com/DataDog/integrations-core/pull/10068

@dim this can be closed, confirmed working (v2 check works good + doesn’t fail on exemplars).