Untyped prometheus metrics not reported

Agent 6.12.1 - Commit: 8ded3a9 - Serialization version: 4.7.1, Running Amazon Linux 2

datadog.yaml:

process_config:
  enabled: "true"
collect_ec2_tags: true
use_dogstatsd: true
dogstatsd_port: 8125
bind_host: localhost
disable_file_logging: true
api_key: ******************
logs_enabled: true

prometheus.d/conf.yaml:

nstances:
  - prometheus_url: http://localhost:8080/metrics
    namespace: "pulsar"
    metrics:
      - "*"
    health_service_check: true

Steps to reproduce the issue:

1. Configure a prometheus endpoint to scrape without any type overrides
2. Run datadog agent

Describe the results you received: Datadog agent no longer reports metrics with no type as “gauge”. This was not the behavior in previous versions of the agent.

Describe the results you expected: Metrics without a type to appear in the dashboard.

Additional information you deem important (e.g. issue happens only occasionally): This https://github.com/DataDog/integrations-core/blob/38a6d70c594a62b9443ed296cc4ee88602d5fc51/prometheus/datadog_checks/prometheus/data/conf.yaml.example#L55 implies that the previous behavior was seen as a bug. “Fixing” this and forcing a type override is a breaking change for many users and presents several problems:

1. It broke our metrics reporting and caused a dangerous condition for support of our product
2. type_overrides seems to not support regex/wildcards. It is unreasonable to expect users with thousands of metrics to scrape to configure all of these one by one.
3. The application that is reporting the metrics may change and start reporting its type. This creates an unnecessary coupling between code and configuration of the agent and assumes that a devops team has specific details of the code base.

[cleanshavenalex]

The Datadog docs should call this out https://www.datadoghq.com/blog/monitor-prometheus-metrics/ "This basic setup includes the Prometheus endpoint, a namespace that will be prepended to all collected metrics, and the metrics you want the Agent to scrape. You can use * wildcards to pull in all metrics that match a given pattern, and you can also map existing Prometheus metrics to custom metric names. "

Unless the metrics are untyped…

[mathewmoon]

@jeremy-lq I think leaving this open would be important for the other users who run into this error as well as providing resolution tracking since users don’t have access to your internal tickets.