Subscribing to unknown topic

In case the topic doesn’t exist and a consumer subscribes to it, there’s no easy way to find out that the topic is not there except making a call to fetch the metadata before attempting to subscribe.

Furthermore, consuming from such a consumer returns ERR_TIMEOUT as if the topic existed, but there were no activity in there.

But the biggest problem is that when the topic gets created and message start flowing there, the consumer that’s subscribed before topic creation will never receive these messages nor would it indicate any issue.

I’m not quite sure how to deal with this. Creating the topic by default with default parameters seem dangerous since it can easily go unnoticed, throwing an error on attempt to subscribe/consume might be a nice feature, kafkacat does that for example. Tracking whether the topic existence have changed and re-subscribing to start receiving messages once the topic was created could also be an option. Not sure which option is better, but silently ignoring the error is definitely worse then any of them.

Would be nice know your opinion on this before deciding how to deal with topic creation in our environment.

[webmakersteve]

librdkafka does have an error code for when the topic does not exist, but I believe this is only done on the producer side rather than the consumer side when using subscriptions. Previously when using the low level consumer, I believe it took a topic object rather than a topic string, and perhaps some more sophisticated validation was done while consuming, but now it is silent about the error.

What I would recommend doing is reading the metadata fetched by default at connect time (it is passed into the callback for connect, or the on('ready'). event. It will give a list of topics that are available.

Then, in your code, you can check that a subscription is subscribing to a topic that you have metadata for, and throw otherwise.

Perhaps this functionality could be encapsulated into the library for a better experience. But I’m undecided. It definitely is annoying that it would fail silently for you.

I’m pretty sure Kafkacat does something similar under the hood. It fetches metadata for the topic you specify with -t and the metadata request comes back with topic does not exist. Would love to hear thoughts on the direction people are most fond of.

[webmakersteve]

I’m happy with it behaving the way @edenhill describes, so I’m going to close the issue 😃