Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Consumer not connecting for `partitions_for_topic()`

See original GitHub issue

Creating a consumer and calling partitions_for_topic() returns None. However calling topics() and then partitions_for_topic() returns the partitions.

    consumer = KafkaConsumer(**myconfig)
    showpartitions(consumer, 'mytopic')
    # returns None

    showtopics(consumer)
    # connects and shows topics

    showpartitions(consumer, 'mytopic')
    # shows partitions

output

WARNING:kafka.coordinator.consumer:group_id is None: disabling auto-commit.
ERROR:main:Failed to fetch partitions for topic.
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=XXX.XXX.XXX.XXX:9443 <connecting> [IPv4 ('XXX.XXX.XXX.XXX', 9443)]>: Loading SSL CA from ae_kafka/all.pem
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=XXX.XXX.XXX.XXX:9443 <connecting> [IPv4 ('XXX.XXX.XXX.XXX', 9443)]>: connecting to XXX.XXX.XXX.XXX:9443 [('XXX.XXX.XXX.XXX', 9443) IPv4]
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=XXX.XXX.XXX.XXX:9443 <authenticating> [IPv4 ('XXX.XXX.XXX.XXX', 9443)]>: Authenticated as myuser via PLAIN
INFO:kafka.conn:<BrokerConnection node_id=bootstrap host=XXX.XXX.XXX.XXX:9443 <authenticating> [IPv4 ('XXX.XXX.XXX.XXX', 9443)]>: Connection complete.
INFO:main:Topics: mytopic, othertopic
INFO:main:Partitions: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39

$ python --version
Python 3.7.2

$ pip install git+https://github.com/dpkp/kafka-python.git@master#egg=kafka-python
Collecting kafka-python from git+https://github.com/dpkp/kafka-python.git@master#egg=kafka-python
  Cloning https://github.com/dpkp/kafka-python.git (to revision master) to /private/var/folders/w9/d60536290678zpvzsc_t04fc0000gn/T/pip-install-fu97thnp/kafka-python
Building wheels for collected packages: kafka-python
  Building wheel for kafka-python (setup.py) ... done
  Stored in directory: /private/var/folders/w9/d60536290678zpvzsc_t04fc0000gn/T/pip-ephem-wheel-cache-heuvanvn/wheels/dd/96/90/dafbe2ada62bf4e6237805d8cd793c2821378d9c7716a67959
Successfully built kafka-python
Installing collected packages: kafka-python
Successfully installed kafka-python-1.4.6.dev0

Issue Analytics

State:
Created 5 years ago
Reactions:1
Comments:6 (2 by maintainers)

Top GitHub Comments

1reaction

jeffwidmancommented, Feb 11, 2020

See the discussion in the PR that modified it: https://github.com/dpkp/kafka-python/pull/1781

Next time, if you want a faster answer, you can always pull up git blame to see the history of the code change and if there was discussion on the commit/PR that modified the behavior.

0reactions

bebosudocommented, Jan 13, 2020

Hi, I was bitten by this problem too today. What’s the reason to not update metadata on each call? Is this to be consistent with the behavior of the java client, or something similar? What about an update_metadata=bool flag in partitions_for_topic in order to force a metadata refresh?

Top Results From Across the Web

Kafka-python get number of partitions for topic - Stack Overflow

I want to get the number of partitions for a specified topic. How do I do that? python · python-2.7 · metadata ·...

A Kafka broker is gracefully shutdown, and incorrect metadata ...

A Kafka broker is gracefully shutdown, and incorrect metadata was passed to the Kafka connect client. Status: Assignee: Priority: Resolution:.

Diagnose and Debug Apache Kafka Issues - Confluent

Rebalancing comes into play in Apache Kafka® when consumers join or leave a consumer group. In either case, there is a different number...

Java Examples for org.apache.kafka.common.PartitionInfo

isEmpty()) { throw new IllegalArgumentException("no partitions for topic: " + ... getName(), null)) { final List<PartitionInfo> partitionInfos = consumer.

Chapter 4. Kafka Consumers: Reading Data from Kafka

Keep in mind that there is no point in adding more consumers than you have partitions in a ... The subcribe() method takes...