Support for kube-rbac-proxy

I have kube-state-metrics deployed with kube-rbac-proxy (https://github.com/brancz/kube-rbac-proxy) for security reasons. 

Datadog needs to be able to authenticate via an RBAC token and TLS certs. 

Additional environment details (Operating System, Cloud provider, etc): Minikube and AWS EKS.

Steps to reproduce the issue:

1. Deploy kube-state-metrics with kube-rbac-proxy
2. Deploy latest datadog agent
3. Configure kubernetes_state to collect from the kube-rbac-proxy port (usually 8443). I had trouble doing this via annotations so i mounted a configmap and overwrote auto_conf.yaml
4. Ensure the service account datadog is using has sufficient privileges.
5. Check datadog agent status for tls error.
6. Exec into the datadog pod and run:

curl -vvv --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer `cat /var/run/secrets/kubernetes.io/serviceaccount/token`" https://172.17.0.8:8443/metrics --output -

Describe the results you received: kubernetes_state (4.1.0) ------------------------ Instance ID: kubernetes_state:1e8ba236f21c65cd [ERROR] Total Runs: 6 Metric Samples: Last Run: 0, Total: 0 Events: Last Run: 0, Total: 0 Service Checks: Last Run: 0, Total: 0 Average Execution Time : 406ms Error: HTTPSConnectionPool(host=‘172.17.0.8’, port=8443): Max retries exceeded with url: /metrics (Caused by SSLError(SSLError(1, u’[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)‘),)) Traceback (most recent call last): File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py”, line 387, in run self.check(copy.deepcopy(self.instances[0])) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/kubernetes_state/kubernetes_state.py”, line 99, in check self.process(scraper_config, metric_transformers=self.METRIC_TRANSFORMERS) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py”, line 294, in process for metric in self.scrape_metrics(scraper_config): File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py”, line 262, in scrape_metrics response = self.poll(scraper_config) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py”, line 411, in poll response = self.send_request(endpoint, scraper_config, headers) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/openmetrics/mixins.py”, line 474, in send_request cert=cert, verify=verify, auth=auth) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py”, line 75, in get return request(‘get’, url, params=params, **kwargs) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/api.py”, line 60, in request return session.request(method=method, url=url, **kwargs) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py”, line 533, in request resp = self.send(prep, **send_kwargs) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/sessions.py”, line 646, in send r = adapter.send(request, **kwargs) File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/adapters.py”, line 514, in send raise SSLError(e, request=request) SSLError: HTTPSConnectionPool(host=‘172.17.0.8’, port=8443): Max retries exceeded with url: /metrics (Caused by SSLError(SSLError(1, u’[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)'),))

Describe the results you expected: Datadog successfully scrapes kube-state-metrics with kube-rbac-proxy.

[AlexisDuf]

Hello,

If you haven’t found a solution yet you can try this. You need to overload kubernetes_state configuration as below:

    ad_identifiers:
      - kube-state-metrics
    init_config:
    instances:
      - kube_state_url: https://%%host%%:8443/metrics
        bearer_token_auth: true
        bearer_token_path: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl_verify: false

kubernetes_state use OpenMetrics which provide such kind of configuration. It works fine for me

@NicklasWallgren yes, as I described above you only need to overload the configuration for kube-state-metrics

[micahlmartin]

Don’t go stale. We definitely need this.