[redisdb] checks autodiscovery fail

Summary Datadog agent does not auto discover the redisdb checks when doing auto discovery by pod annotation with instance config password: "%%env_REDIS_PASSWORD%%"

Steps to reproduce the issue: Kubernetes Cluster version: 1.16.9 Datadog agent Chart version: 2.0.4

1. Install the datadog daemonset using this helm chart https://github.com/helm/charts/tree/master/stable/datadog
2. Install redis by using this helm chart https://github.com/bitnami/charts/tree/master/bitnami/redis and supply the below datadog discovery annotations through helm value files

master.podAnnotations: {
    ad.datadoghq.com/redis.check_names: '["redisdb"]'
    ad.datadoghq.com/redis.init_configs: '[{}]'
    ad.datadoghq.com/redis.instances: |
      [
        {
          "host": "%%host%%",
          "port":"6379",
          "password":"%%env_REDIS_PASSWORD%%"
        }
      ]
 }

1. use kubectx exec -it <POD_NAME> sh to execute into the shell of the datadog agent on the same node as the redis pod
2. execute agent status command

Actual Result: I cannot see and redis check info display on the agent status

Expected: If the auto discover successful I must be able to see that the agent discover the redis pod

Additional information:

1. Although i also try to use agent configcheck command but still could not see any information regard to the redis check

2. However, when i try to hard code the redis password to the annotation for example:

master.podAnnotations: {
    ad.datadoghq.com/redis.check_names: '["redisdb"]'
    ad.datadoghq.com/redis.init_configs: '[{}]'
    ad.datadoghq.com/redis.instances: |
      [
        {
          "host": "%%host%%",
          "port":"6379",
          "password":"123456"
        }
      ]
 }

Suddenly the auto discovery of the redis check work and the information of the check/discovery appear when I execute agent status and agent cofigcheck

[vboulineau]

@chan43999 That’s not possible (except if you pass REDIS_PASSWORD in plain text!), secret in PODs are (hopefully) not expanded when querying the Kubelet or APIServer for the POD object. Instead we will get:

                            {
                                "name": "REDIS_PASSWORD",
                                "valueFrom": {
                                    "secretKeyRef": {
                                        "key": "password",
                                        "name": "redis-password"
                                    }
                                }
                            },

Which won’t allow the agent to fetch the value

[chan43999]

@ahmed-mez @vboulineau Got it. Thank you for your explanation and suggestion. The secret management feature look good to me. I will go ahead a try it. Cheers!!!