Iterator interface and poll block with two consumers reading from one topic

I have a topic with a single partition and two KafkaConsumer in two separate processes with different group_id. When I try to get messages (there are none yet) from this topic either using iterator interface (with consumer_timeout_ms=10) or by using KafkaConsumer.poll, the consumer blocks until session_timeout_ms expires, although I believe it’s supposed to block for at most consumer_timeout_ms. Is this assumption correct? What might be causing it to block for longer? If there is only one reader, or if group_id is None, everything is working as expected. Here are log messages before consumer blocks:

2017-09-19 15:01:06,706 [INFO] cluster: Group coordinator for my-group is BrokerMetadata(nodeId=0, host='kafka', port=9092, rack=None)
2017-09-19 15:01:06,707 [INFO] base: Discovered coordinator 0 for group my-group
2017-09-19 15:01:06,707 [INFO] consumer: Revoking previously assigned partitions set() for group my-group
2017-09-19 15:01:06,708 [INFO] base: (Re-)joining group my-group

Edit: In case debug logs are useful, here are they:

2017-09-19 18:41:55,821 [DEBUG] base: Sending group coordinator request for group my-group to broker 0
2017-09-19 18:41:55,822 [DEBUG] conn: <BrokerConnection node_id=0 host=kafka/172.23.0.2 port=9092> Request 3: GroupCoordinatorRequest_v0(consumer_group='my-group')
2017-09-19 18:41:55,822 [DEBUG] client_async: Sending metadata request MetadataRequest_v1(topics=['my-topic']) to node 0
2017-09-19 18:41:55,823 [DEBUG] conn: <BrokerConnection node_id=0 host=kafka/172.23.0.2 port=9092> Request 4: MetadataRequest_v1(topics=['my-topic'])
2017-09-19 18:41:55,823 [DEBUG] consumer: Cannot auto-commit offsets for group my-group because the coordinator is unknown
2017-09-19 18:41:55,823 [DEBUG] conn: <BrokerConnection node_id=0 host=kafka/172.23.0.2 port=9092> Response 3: GroupCoordinatorResponse_v0(error_code=0, coordinator_id=0, host='kafka', port=9092)
2017-09-19 18:41:55,824 [DEBUG] base: Received group coordinator response GroupCoordinatorResponse_v0(error_code=0, coordinator_id=0, host='kafka', port=9092)
2017-09-19 18:41:55,824 [DEBUG] cluster: Updating coordinator for my-group: GroupCoordinatorResponse_v0(error_code=0, coordinator_id=0, host='kafka', port=9092)
2017-09-19 18:41:55,824 [INFO] cluster: Group coordinator for my-group is BrokerMetadata(nodeId=0, host='kafka', port=9092, rack=None)
2017-09-19 18:41:55,824 [INFO] base: Discovered coordinator 0 for group my-group
2017-09-19 18:41:55,825 [DEBUG] conn: <BrokerConnection node_id=0 host=kafka/172.23.0.2 port=9092> Response 4: MetadataResponse_v1(brokers=[(node_id=0, host='kafka', port=9092, rack=None)], controller_id=0, topics=[(error_code=0, topic='my-topic', is_internal=False, partitions=[(error_code=0, partition=0, leader=0, replicas=[0], isr=[0])])])
2017-09-19 18:41:55,825 [DEBUG] cluster: Updated cluster metadata to ClusterMetadata(brokers: 1, topics: 1, groups: 1)
2017-09-19 18:41:55,825 [INFO] consumer: Revoking previously assigned partitions set() for group my-group
2017-09-19 18:41:55,826 [INFO] base: (Re-)joining group my-group
2017-09-19 18:41:55,826 [DEBUG] base: Sending JoinGroup (JoinGroupRequest_v0(group='my-group', session_timeout=30000, member_id='', protocol_type='consumer', group_protocols=[(protocol_name='range', protocol_metadata=b'\x00\x00\x00\x00\x00\x01\x00\x0fmy-topic\x00\x00\x00\x00'), (protocol_name='roundrobin', protocol_metadata=b'\x00\x00\x00\x00\x00\x01\x00\x0fmy-topic\x00\x00\x00\x00')])) to coordinator 0
2017-09-19 18:41:55,827 [DEBUG] conn: <BrokerConnection node_id=0 host=kafka/172.23.0.2 port=9092> Request 5: JoinGroupRequest_v0(group='my-group', session_timeout=30000, member_id='', protocol_type='consumer', group_protocols=[(protocol_name='range', protocol_metadata=b'\x00\x00\x00\x00\x00\x01\x00\x0fmy-topic\x00\x00\x00\x00'), (protocol_name='roundrobin', protocol_metadata=b'\x00\x00\x00\x00\x00\x01\x00\x0fmy-topic\x00\x00\x00\x00')])
2017-09-19 18:41:55,924 [DEBUG] consumer: No offsets to commit
2017-09-19 18:41:55,925 [DEBUG] consumer: Successfully auto-committed offsets for group my-group

[lopuhin]

Thanks for looking into it @tvoinarovskyi ! I tried to build a reproducible example but didn’t 100% succeed so far - I only managed to “reproduce” it when I don’t close the consumer (as you said here https://github.com/dpkp/kafka-python/issues/1223#issuecomment-330829458).

I’m using Python 3.5 and Ubuntu 16.04.

Here is issue_1223.py:

import logging

import kafka
import kafka.client


topic_name = 'my-topic'
group_name = 'my-group'


def main():
    logging.basicConfig(
        level=logging.INFO,
        format='%(asctime)s [%(levelname)s] %(module)s: %(message)s')

    logging.info('Creating topic')
    client = kafka.client.KafkaClient()
    topics = client.cluster.topics()
    if topic_name not in topics:
        client.add_topic(topic_name)
    logging.info('done')

    consumer = kafka.KafkaConsumer(
        topic_name,
        group_id=group_name,
        consumer_timeout_ms=1000,
    )
    try:
        while True:
            logging.info('reading consumer...')
            for _ in consumer:
                pass
            logging.info('done')
    except KeyboardInterrupt:
        logging.info('Ctrl+C pressed, closing consumer and exiting')
        consumer.close()


if __name__ == '__main__':
    main()

I start kafka + zookeeper like this:

docker run -it --rm --name kafka -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=127.0.0.1 --env ADVERTISED_PORT=9092 spotify/kafka

Then if I start python issue_1223.py in one tab, and the same in the other tab, everything is working fine. If I then remove the last line from main (consumer.close()), and restart one of consumers, I get the hang up.

To sum up: I’ll try to understand why did I get the issue in my original code, where I think I didn’t have any consumers exiting, will post an update if I understand it.

[tvoinarovskyi]

Could you provide a reproducible example, I fail to reproduce this on my local machine… It’s consuming just fine for me.