kafka.common.KafkaTimeoutError: ('Failed to update metadata after %s secs.', 60.0)

kafka version: 0.8.2.0-1.kafka1.3.2.p0.15 (cloudera released)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/archimonde/lib/python2.6/site-packages/kafka/producer/kafka.py", line 357, in send
    self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0)
  File "/opt/archimonde/lib/python2.6/site-packages/kafka/producer/kafka.py", line 465, in _wait_on_metadata
    "Failed to update metadata after %s secs.", max_wait)
kafka.common.KafkaTimeoutError: ('Failed to update metadata after %s secs.', 60.0)

But it’s ok on 2.0.0-1.kafka2.0.0.p0.12.

[nicholasserra]

Also wondering fix. I see this error when running kafka via Docker but not when installing from binary.

EDIT: So wondering what the root cause may be to help me pinpoint issues between environments.

[griff122]

Another cause for this issue is using a bytes object as the topic name, instead of a string. In python 3, you could have a string come in as a bytes object (a = b'test_string'). If this happens, you can just convert the topic name to a utf-8 string, and it might start working. It worked for me.

if type(kafka_topic) == bytes:
    kafka_topic = kafka_topic.decode('utf-8')
kafka_producer = KafkaProducer(bootstrap_servers=kafka_brokers)
kafka_producer.send(kafka_topic, payload)
kafka_producer.flush()