question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Help: KafkaConsumer iterator ignores consumer_timeout_ms if entire cluster is down

See original GitHub issue

Hi! Forgive my poor English, but I need your help. @dpkp While stress testing Kafka, I have found that comsumer can block forever when all brokers down. After all brokers down, comsumer will fall into a loop to getting metadata from broker. This loop will never break util one or more brokers restart.
Is there a way that consumer can break this loop? I do not want block here, I want to do some other things when brokers down.

my code is like this:

consumer = KafkaConsumer(bootstrap_servers=['xxxxx','xxxxx'],
                                              group_id="test1",
                                              consumer_timeout_ms= 2000)
partition = TopicPartition("topic1", 0)
consumer.assign([partition])
consumer.seek_to_end()
for message in consumer: # it will block here when all brokers down
    print msg

After consumer block, I got this when keyboardInterrupt: image

other infos: kafka_python-1.3.2 python2.6.3 kafka_2.11-0.10.2.1

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:2
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
stigokcommented, Jan 30, 2018

I experience the same thing. And the loop makes CPU go to 100%.

0reactions
JamesMackerelcommented, Mar 8, 2018

After testing, I found a surprise. It will block, until the cluster is back. But I still want to know if there is any way to know if the cluster is down in consumer.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Chapter 4. Kafka Consumers: Reading Data from Kafka
Kafka consumers are typically part of a consumer group . When multiple consumers are subscribed to a topic and belong to the same...
Read more >
KafkaConsumer (kafka 2.5.0 API)
A client that consumes records from a Kafka cluster. This client transparently handles the failure of Kafka brokers, and transparently adapts as topic ......
Read more >
Kafka 3.3 Documentation
Servers: Kafka is run as a cluster of one or more servers that can span multiple ... The default value for the consumer...
Read more >
KafkaConsumer (kafka 0.11.0.2 API)
A client that consumes records from a Kafka cluster. ... All consumer instances sharing the same group.id will be part of the same...
Read more >
Documentation - Apache Kafka
Kafka MirrorMaker provides geo-replication support for your clusters. ... If all the consumer instances have the same consumer group, then the records will ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found