Deactivate auto reconnect for consumer

Here is the problem

Sample code: use the next iterator, and go through. I handle my own timeouts, retries, what I do, when I commit (one message at a time), what I log, etc.

from kafka import KafkaConsumer
consumer = KafkaConsumer('mytopic', boostrap_servers='myserver', consumer_timeout_ms=1000, enable_auto_commit=False)

while True:
  try:
    print "start iteration"
    data = next(consumer)
    # do some processing, and if it went well, commit
    consumer.commit()
  except StopIteration:
    print "cycle back"

For normal processing, when kafka is running fine, it will print start interation, cycle back every second if I don’t have any new messages in my kafka topic, except that if I stop kafka, it will block at the next(consumer) forever. And then if I restart kafka, it will reconnect and do as if nothing had gone wrong.

How can I stop the “magic” behavior? I’d like it to just throw an exception if it disconnects, and let me do the reconnect on my own terms.

[felipou]

Or an error event. If the client can’t connect to any broker at all, I’d like to know that. Perhaps there is a network issue on something, and I’d like to issue a warning so that someone can take action, or maybe just be aware that the client isn’t able to connect and possibly delayed.

[jeffwidman]

I’d like for this to exist as well, in containerized environments like kubernetes, if the cluster goes down and comes back up, the brokers will get different IPs, and the client will keep trying to refresh metadata from nonexistent brokers forever, with a timeout or a number of maximum retries, the client would die and kubernetes would fire it again.

Although unrelated to the original issue, that is a definite bug, we shouldn’t cache those IP’s forever. However, IIRC since this issue was last updated there have been a number of fixes to DNS resolution that have been merged.

I’m going to close this. If you can replicate this issue with the 2.x series, please open a new ticket and lets get it fixed.