Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Consumer reassignment to Offset.End does not work as expected

See original GitHub issue

Description

When consumer is assigned to the end of partition then unassigned and after some time again assigned to the end of partition, it will be consuming from the offset at which it was unassigned, while the expected behaviour is to consume from the end of partition.

How to reproduce

var producer =
    new ProducerBuilder<Null, string>(
            new ProducerConfig
            {
                BootstrapServers = kafkaBrokers,
                Acks = Acks.None
            })
        .Build();

Task.Run(
    async () =>
    {
        while (true)
        {
            producer.Produce(
                testTopic,
                new Message<Null, string>
                {
                    Value = DateTime.UtcNow.ToString()
                });
            await Task.Delay(1000);
        }
    });

var consumer =
    new ConsumerBuilder<Null, string>(
            new ConsumerConfig
            {
                GroupId = Guid.NewGuid().ToString(),
                BootstrapServers = kafkaBrokers
            })
        .Build();

Task.Run(
    () =>
    {
        while (true)
        {
            var consumeResult = consumer.Consume();
            logger.Info($"Consumed: {consumeResult.Value}");
        }
    });

consumer.Assign(new TopicPartitionOffset(testTopic, Offset.End));
logger.Info("Subscribed.");

Task.Delay(TimeSpan.FromSeconds(5)).ContinueWith(t =>
{
    consumer.Assign(Enumerable.Empty<TopicPartition>());
    logger.Info("Unsubscribed.");
});

Task.Delay(TimeSpan.FromSeconds(10)).ContinueWith(t =>
{
    consumer.Assign(new TopicPartitionOffset(testTopic, Offset.End));
    logger.Info("Subscribed.");
});

Task.Delay(TimeSpan.FromSeconds(15)).ContinueWith(t =>
{
    consumer.Assign(Enumerable.Empty<TopicPartition>());
    logger.Info("Unsubscribed.");
});

Actual result:

2020-04-18 13:27:29,853 INFO  [1]: Log - Subscribed.
2020-04-18 13:27:31,115 INFO  [5]: Log - Consumed: 18.04.2020 13:27:30
2020-04-18 13:27:32,159 INFO  [5]: Log - Consumed: 18.04.2020 13:27:31
2020-04-18 13:27:33,209 INFO  [5]: Log - Consumed: 18.04.2020 13:27:32
2020-04-18 13:27:34,111 INFO  [5]: Log - Consumed: 18.04.2020 13:27:33
2020-04-18 13:27:34,863 INFO  [4]: Log - Unsubscribed.
2020-04-18 13:27:39,870 INFO  [4]: Log - Subscribed.
2020-04-18 13:27:39,999 INFO  [5]: Log - Consumed: 18.04.2020 13:27:34
2020-04-18 13:27:40,000 INFO  [5]: Log - Consumed: 18.04.2020 13:27:35
2020-04-18 13:27:40,000 INFO  [5]: Log - Consumed: 18.04.2020 13:27:36
2020-04-18 13:27:40,000 INFO  [5]: Log - Consumed: 18.04.2020 13:27:37
2020-04-18 13:27:40,000 INFO  [5]: Log - Consumed: 18.04.2020 13:27:38
2020-04-18 13:27:40,130 INFO  [5]: Log - Consumed: 18.04.2020 13:27:39
2020-04-18 13:27:41,182 INFO  [5]: Log - Consumed: 18.04.2020 13:27:40
2020-04-18 13:27:42,231 INFO  [5]: Log - Consumed: 18.04.2020 13:27:41
2020-04-18 13:27:43,133 INFO  [5]: Log - Consumed: 18.04.2020 13:27:42
2020-04-18 13:27:44,181 INFO  [5]: Log - Consumed: 18.04.2020 13:27:43
2020-04-18 13:27:44,870 INFO  [8]: Log - Unsubscribed.

Expected result:

2020-04-18 13:27:29,853 INFO  [1]: Log - Subscribed.
2020-04-18 13:27:31,115 INFO  [5]: Log - Consumed: 18.04.2020 13:27:30
2020-04-18 13:27:32,159 INFO  [5]: Log - Consumed: 18.04.2020 13:27:31
2020-04-18 13:27:33,209 INFO  [5]: Log - Consumed: 18.04.2020 13:27:32
2020-04-18 13:27:34,111 INFO  [5]: Log - Consumed: 18.04.2020 13:27:33
2020-04-18 13:27:34,863 INFO  [4]: Log - Unsubscribed.
2020-04-18 13:27:39,870 INFO  [4]: Log - Subscribed.
2020-04-18 13:27:40,130 INFO  [5]: Log - Consumed: 18.04.2020 13:27:39
2020-04-18 13:27:41,182 INFO  [5]: Log - Consumed: 18.04.2020 13:27:40
2020-04-18 13:27:42,231 INFO  [5]: Log - Consumed: 18.04.2020 13:27:41
2020-04-18 13:27:43,133 INFO  [5]: Log - Consumed: 18.04.2020 13:27:42
2020-04-18 13:27:44,181 INFO  [5]: Log - Consumed: 18.04.2020 13:27:43
2020-04-18 13:27:44,870 INFO  [8]: Log - Unsubscribed.

Issue Analytics

State:
Created 3 years ago
Comments:13 (7 by maintainers)

Top GitHub Comments

1reaction

mhowlettcommented, May 7, 2020

the issue is here (identified by @edenhill): https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_offset.c#L793 - the cached offset is being used when this isn’t valid. expect a fix in 1.5.

1reaction

mhowlettcommented, May 7, 2020

thanks for following up - https://github.com/edenhill/librdkafka/issues/2782 is fixed in 1.4.2 and unfortunately i can still replicate using the Consumer_Assign2 test above. fyi: @edenhill

Top Results From Across the Web

Kafka Consumer not consuming from last commited offset ...

I have a consumer polling from subscribed topic. It consumes each message and does some processing (within seconds), pushes to different topic ...

Unable to reset offset for a single partition - Java Clients

offset.reset , I'm trying to explicitly reset the offset for each consumer, when it gets a state partition assigned. I tried calling ....

How to | Conduktor Docs

Go to the Consumer Group itself, and Reset Offsets from here. How to reassign the partitions of a Topic?. In the Topic Details...

KafkaConsumer (kafka 2.2.0 API)

A client that consumes records from a Kafka cluster. This client transparently handles the failure of Kafka brokers, and transparently adapts as topic ......

KafkaConsumer — kafka-python 2.0.2-dev documentation

The last offset of a partition is the offset of the upcoming message, i.e. the offset of the last available message + 1....