Consuming from Invalid Topic and End of Valid Topic is Indistinguishable

Description

Subscribing to a valid or invalid topic and calling var messageConsumed = Consumer.Consume(out kafkaMessage, pollInterval) the consumer just returns a null value making it unclear if the consumer is just reading from the end of a valid topic and there is no more messages or if the consumer is subscribed to an invalid topic.

Observed behavior is consuming from an invalid topic silently returns a null message.

Expected behavior for the case of an invalid topic was that it would be similar to what is displayed by producer: Trying to consume from an invalid topic would result in a Message being returned with invalid offset and error code such as code 3 for Unknown Topic or Partition, or at the very least and exception being thrown for consuming from invalid topic.

Shouldn’t there be a clear difference between consuming from the end of a valid topic and a topic that doesn’t exist so the implementation knows that it is in an error state?

As a side note it occurred to me to check for the OnError or OnConsumeError events while checking for any similar issues and neither of these events are being triggered in the invalid topic scenario.

How to reproduce

Just consuming from a none existent topic as follows:

try
{
    var config  = new Dictionary<string, object>
    {
        {"group.id", "ConsumerId"},
        {"enable.auto.commit", false},
        {"log.connection.close", false},
        {"socket.keepalive.enable", true},
        {"statistics.interval.ms", 60000},
        {"bootstrap.servers", localhost:9092},
    };
    var consumer= new Consumer(config);
    consumer.Subscribe("Some-Invalid-Topic-Name");
    var messageConsumed = false;
    while (!messageConsumed && kafkaMessage == null)
    {
        messageConsumed = consumer.Consume(out kafkaMessage, pollInterval);
    }
    Console.WriteLine("Never Escapes the While Loop.");
}
catch (exception e)
{
    Console.WriteLine("Never Catches a Exception.");
}

Currently using Kafka Server version 2.11-0.10.2.1 Server Configured to Auto Create Topics = false

Confluent Kafka Dotnet version 11.2

Passed Consumer Configuration:

• group.id was set
• enable.auto.commit = false
• log.connection.close = false
• socket.keepalive.enable = true
• statistics.interval.ms = 60000
• bootstrap.servers pointed to network cluster

Currently developing Windows 7 / 10 / Server 2012 operating systems we are moving towards .net core 2.0 and supporting Linux Centos/Ubuntu distros as well.

Checklist

Please provide the following information:

• Confluent.Kafka nuget version:
• Apache Kafka version:
• Client configuration:
• Operating system:
• Provide logs (with “debug” : “…” as necessary in configuration)
• Provide broker log excerpts
• Critical issue

[edenhill]

I agree, it should fire off an _UNKNOWN_TOPIC or …_PARTITION consumer error.

This issue should be moved to librdkafka.

[mhowlett]

closing, this is being tracked in https://github.com/edenhill/librdkafka/issues/1540