Help: KafkaConsumer iterator ignores consumer_timeout_ms if entire cluster is down
See original GitHub issueHi!
Forgive my poor English, but I need your help. @dpkp
While stress testing Kafka, I have found that comsumer can block forever when all brokers down. After all brokers down, comsumer will fall into a loop to getting metadata from broker. This loop will never break util one or more brokers restart.
Is there a way that consumer can break this loop? I do not want block here, I want to do some other things when brokers down.
my code is like this:
consumer = KafkaConsumer(bootstrap_servers=['xxxxx','xxxxx'],
group_id="test1",
consumer_timeout_ms= 2000)
partition = TopicPartition("topic1", 0)
consumer.assign([partition])
consumer.seek_to_end()
for message in consumer: # it will block here when all brokers down
print msg
After consumer block, I got this when keyboardInterrupt:
other infos: kafka_python-1.3.2 python2.6.3 kafka_2.11-0.10.2.1
Issue Analytics
- State:
- Created 6 years ago
- Reactions:2
- Comments:5 (1 by maintainers)
Top Results From Across the Web
Chapter 4. Kafka Consumers: Reading Data from Kafka
Kafka consumers are typically part of a consumer group . When multiple consumers are subscribed to a topic and belong to the same...
Read more >KafkaConsumer (kafka 2.5.0 API)
A client that consumes records from a Kafka cluster. This client transparently handles the failure of Kafka brokers, and transparently adapts as topic ......
Read more >Kafka 3.3 Documentation
Servers: Kafka is run as a cluster of one or more servers that can span multiple ... The default value for the consumer...
Read more >KafkaConsumer (kafka 0.11.0.2 API)
A client that consumes records from a Kafka cluster. ... All consumer instances sharing the same group.id will be part of the same...
Read more >Documentation - Apache Kafka
Kafka MirrorMaker provides geo-replication support for your clusters. ... If all the consumer instances have the same consumer group, then the records will ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I experience the same thing. And the loop makes CPU go to 100%.
After testing, I found a surprise. It will block, until the cluster is back. But I still want to know if there is any way to know if the cluster is down in consumer.