Kafka consumer integration not collecting metrics with datadog-agent v6.4.2

Output of the info page (if this is a bug)

Getting the status from the agent.

==============
Agent (v6.4.2)
==============

  Status date: 2018-08-31 08:30:33.349557 UTC
  Pid: 3390
  Python Version: 2.7.12
  Logs:
  Check Runners: 2
  Log Level: WARN

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -0.001602947 s
    System UTC time: 2018-08-31 08:30:33.349557 UTC

  Host Info
  =========
    bootTime: 2018-08-30 16:38:55.000000 UTC
    kernelVersion: 4.4.0-1065-aws
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 16.04
    procs: 203
    uptime: 104
    virtualizationRole: guest
    virtualizationSystem: xen

  Hostnames
  =========
    ec2-hostname: ip-10-0-10-106.ec2.internal
    hostname: i-03cf5077d74ea9b5c
    instance-id: i-03cf5077d74ea9b5c
    socket-fqdn: ip-10-0-10-106.ec2.internal.
    socket-hostname: ip-10-0-10-106

=========
Collector
=========

  Running Checks
  ==============
    cpu
    ---
      Total Runs: 3799
      Metric Samples: 6, Total: 22788
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 0ms


    disk (1.2.0)
    ------------
      Total Runs: 3799
      Metric Samples: 140, Total: over 100K
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 43ms


    file_handle
    -----------
      Total Runs: 3799
      Metric Samples: 1, Total: 3799
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 0ms


    io
    --
      Total Runs: 3799
      Metric Samples: 104, Total: over 100K
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 9ms


    kafka_consumer (1.4.0)
    ----------------------
      Total Runs: 3799
      Metric Samples: 0, Total: 0
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 84ms


    load
    ----
      Total Runs: 3799
      Metric Samples: 6, Total: 22794
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 0ms


    memory
    ------
      Total Runs: 3799
      Metric Samples: 17, Total: 64583
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 0ms


    network (1.6.0)
    ---------------
      Total Runs: 3799
      Metric Samples: 20, Total: 75980
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 0ms


    ntp
    ---
      Total Runs: 3799
      Metric Samples: 1, Total: 3738
      Events: 0, Total: 0
      Service Checks: 1, Total: 3799
      Average Execution Time : 0ms


    uptime
    ------
      Total Runs: 3799
      Metric Samples: 1, Total: 3799
      Events: 0, Total: 0
      Service Checks: 0, Total: 0
      Average Execution Time : 0ms


========
JMXFetch
========

  Initialized checks
  ==================
    kafka
      instance_name : kafka-localhost-9997
      message :
      metric_count : 44
      service_check_count : 0
      status : OK
  Failed checks
  =============
    no checks

=========
Forwarder
=========

  CheckRunsV1: 3799
  Dropped: 0
  DroppedOnInput: 0
  Errors: 0
  Events: 0
  HostMetadata: 0
  IntakeV1: 288
  Metadata: 0
  Requeued: 0
  Retried: 0
  RetryQueueSize: 0
  Series: 0
  ServiceChecks: 0
  SketchSeries: 0
  Success: 7886
  TimeseriesV1: 3799

  API Keys status
  ===============
    https://6-4-2-app.agent.datadoghq.com,*************************a1107: API Key valid

==========
Logs Agent
==========

  kafka
  -----
    Type: file
    Path: /data1/kafka-logs/server.log
    Status: OK
    Inputs: /data1/kafka-logs/server.log

=========
DogStatsD
=========

  Checks Metric Sample: 1.162764e+06
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 3799
  Series Flushed: 1.219162e+06
  Service Check: 41833
  Dogstatsd Metric Sample: 190896

Describe what happened:

I upgraded from datadog-agent from v5.25.1 v6.4.2 on an Ubuntu xenial distro using debs from Datadog apt repository. I updated my old datadog-agent config data to the new structure, e.g., /etc/datadog-agent/datadog.yaml, moving /etc/dd-agent/conf.d/kafka.yaml to /etc/dd-agent/conf.d/kafka.d/conf.yaml and moving /etc/dd-agent/conf.d/kafka_consumer.yaml to /etc/dd-agent/conf.d/kafka_consumer.d/conf.yaml.

The datadog agent is collecting and reporting metrics for the kafka server and also reporting logs, but it’s failing to collect any metrics for kafka_consumer.

A snippet of output from sudo datadog-agent check kafka -l debug will show that it loaded kafka_consumer config properly and also loaded checks for kafka_consumer.:

...
2018-08-31 08:24:56 UTC | DEBUG | (file.go:191 in collectEntry) | Found valid configuration in file: /etc/datadog-agent/conf.d/kafka.d/conf.yaml
2018-08-31 08:24:56 UTC | DEBUG | (file.go:191 in collectEntry) | Found valid configuration in file: /etc/datadog-agent/conf.d/kafka.d/metrics.yaml
2018-08-31 08:24:56 UTC | DEBUG | (file.go:191 in collectEntry) | Found valid configuration in file: /etc/datadog-agent/conf.d/kafka_consumer.d/conf.yaml
...
...
...
2018-08-31 08:24:57 UTC | WARN | (check.go:276 in Configure) | passing `agentConfig` to the constructor is deprecated, please use the `get_config` function from the 'datadog_agent' package (kafka_consumer).
2018-08-31 08:24:57 UTC | DEBUG | (check.go:278 in Configure) | python check configure done kafka_consumer
2018-08-31 08:24:57 UTC | DEBUG | (loader.go:169 in Load) | python loader: done loading check kafka_consumer (version 1.4.0)
2018-08-31 08:24:57 UTC | DEBUG | (scheduler.go:137 in getChecks) | Python Check Loader: successfully loaded check 'kafka_consumer'

FYI, we downgraded our kafka consumers from kafka-python-1.4.2 to kafka-python-1.3.5 because of bugs in v1.4.2. Is a version compatibility between datadog-agent kafka-python consumer and our consumer mandatory for collecting metrics?

Describe what you expected:

kafka_consumer offset metrics should be collected and reported properly.

Steps to reproduce the issue:

• Run kafka_consumer using kafka-python v1.3.5
• Install datadog-agent v6.4.2 in kafka broker server 1.1.0 and enable kafka and kafka_consumer integration

Additional environment details (Operating System, Cloud provider, etc):

• Distribution: Ubuntu Xenial 16.04 LTS on AWS EC2 instance

[dabcoder]

Hello, if you are still experiencing this issue, could you send an email to [e-mail] with those details so that we can follow up? Thanks in advance.

[nutanix-bigbasket]

+1