Agent v6: prometheus summary counts are not sent as monotonic counters

Prometheus check does not send the .count metric generated from summary as deltas, but the raw value is sent. A similar issue was solved for counters with send_monotonic_counter option, but send_monotonic_counter does not have any effect on counters which are part of summary metrics.

Steps to reproduce the issue:

1. Add a summary metric to your prometheus endpoint
2. Set the agent to scrape it
3. Observe that the value is always increasing

Describe the results you received:

A sloped line can be seen in this graph

Describe the results you expected:

Datadog agent should send deltas and and a (more or less) flat line should be visible. It should either respect send_monotonic_counter setting or a newly introduced similar setting (for backward compatibility).

Additional information you deem important (e.g. issue happens only occasionally):

This issue is related to https://github.com/DataDog/integrations-core/issues/1303 where send_monotonic_counter was introduced and is probably also related to https://github.com/DataDog/integrations-core/issues/1303#issuecomment-496142520

[sirianni]

Shouldn’t the histograms _sum also be sent as a monotonic counter? It’s currently hardcoded to send as a gauge. This makes it impossible to divide the _sum by the _count in order to get an average for the distribution (which seems to be the point of sending the sum & count).

https://github.com/DataDog/integrations-core/blob/master/datadog_checks_base/datadog_checks/base/checks/openmetrics/mixins.py#L706-L713

            if sample[self.SAMPLE_NAME].endswith("_sum"):
                tags = self._metric_tags(metric_name, val, sample, scraper_config, hostname=custom_hostname)
                self.gauge(
                    "{}.{}.sum".format(scraper_config['namespace'], metric_name),
                    val,
                    tags=tags,
                    hostname=custom_hostname,
                )

[tonglil]

I agree with @martin-sucha that counters (and sums) collected form summaries metrics should also be sent as monotonic counters, instead of defaulting to the gauge metric type in Datadog upon ingestion.

This should be true with the send_monotonic_counter: true option because when metrics get scraped, I want both counters and summary counts to be monotonic, so it is not possible to set that option to false (which is the requirement for the solution posted above).