[kubernetes_state] Metrics for HPA renamed in 2.x

Output of the info page It may be poor judgement, but it seems to me irrelevant for this issue

Additional environment details (Operating System, Cloud provider, etc): Datadog Agent 7.24.1 with Datadog Cluster Agent 1.9.1 deployed in a Kubernetes 1.17 cluster.

Steps to reproduce the issue:

1. Deploy the newest stable Datadog Agent (optional: and the cluster agent) in a Kubernetes 1.17+ cluster
2. Deploy kube-state-metrics 2.0.0-beta (The only version compatible with Kubernetes 1.17+)
3. Create a deployment with horizontal pod autoscaler (HPA)
4. Look at the metrics explorer/summary in Datadog.

Describe the results you received: The kubernetes_state.hpa.* metrics are not being sent. The reason is that kube-state-metrics 2.x has renamed the metrics internally from kube_hpa_* to kube_horizontalpodautoscaler_* (https://github.com/kubernetes/kube-state-metrics/pull/1003). Same metrics, different prefix. These metrics are not correctly recognized by the kubernetes_state check.

Describe the results you expected: I expect the Datadog integration to send the horizontal pod autoscaler metrics to Datadog.

Additional information you deem important (e.g. issue happens only occasionally): I have worked around the issue by ‘monkey-patching’ the Docker container to rename the kube_hpa_* metrics to kube_horizontalpodautoscaler_* at:

• https://github.com/DataDog/integrations-core/blob/46b278091b74ef822e7a2911bc0e1637edadf9cd/kubernetes_state/datadog_checks/kubernetes_state/kubernetes_state.py#L219-L223
• https://github.com/DataDog/integrations-core/blob/46b278091b74ef822e7a2911bc0e1637edadf9cd/kubernetes_state/datadog_checks/kubernetes_state/kubernetes_state.py#L328
• https://github.com/DataDog/integrations-core/blob/46b278091b74ef822e7a2911bc0e1637edadf9cd/kubernetes_state/datadog_checks/kubernetes_state/kubernetes_state.py#L363

However, this is not strictly compatible with the old kube_state_metrics as kube_state_metrics 1.x will report a label hpa with the name of the horizontal pod autoscaler, while in 2.x this tag is renamed to horizontalpodautoscaler, so I had to change some monitors/dashboards to make it work correctly.

[TheHodge1234]

I have just run into this issue, after Datadog support told me to upgrade to 2.x to fix another issue I have now lost the HPA metrics!

This needs fixing

[TheHodge1234]

I have an open ticket with Datadog support on this - they are pretty adamant that it works - but everything they have told me to do so far doesn’t work!

They appear to believe that the issue is that we are utilising KSM Legacy and not KSM Core - but i have tried (using their info) to move to KSM Core, and it still breaks the HPA metrics. KSM Core does not require us to even run an external KSM pod