question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Proper way to reset offset for a given GroupId

See original GitHub issue

Description

I need to be able to “rewind” a consumer to the beginning of a topic and re-process existing messages. Is this properly supported?

I’m using the following code, but am not getting the expected result.

foreach (var topicPartition in Consumer.Assignment)
{
	Consumer.Assign(new TopicPartitionOffset(topicPartition, Offset.Beginning));
}

I’ve tried both .Seek() and .Assign() in the code above.

What is the proper way of resetting the offset on a per-group basis?

Checklist

  • [1.0.1.1 ] Confluent.Kafka nuget version.

Issue Analytics

  • State:open
  • Created 4 years ago
  • Reactions:5
  • Comments:8 (3 by maintainers)

github_iconTop GitHub Comments

8reactions
russaucommented, Jan 22, 2020

FWIW This appears to be working for me with ver 1.3.0:

var consumer = new ConsumerBuilder<string, string>(consumerConfig)
.SetPartitionsAssignedHandler((c, partitions) =>
{
    var offsets = partitions.Select(tp => new TopicPartitionOffset(tp, Offset.Beginning));
    return offsets;
})
.Build();
....
0reactions
harvzorcommented, Feb 2, 2023

if you don’t want to need to stop your consumers, you’ll need to signal them in some way that they should start reprocessing, then you could use IConsumer.QueryWatermarkOffsets to get the beginning offset for each partition, then seek to that position, then commit as usual after consuming messages.

This appears to be the suggested code:

private void ResetOffset()
{
    // Assume partition 0, good luck finding out what your partition is otherwise!
    var topicPartition = new TopicPartition(_topic, new Partition(0));
    var watermarkOffsets = _consumer.QueryWatermarkOffsets(topicPartition, TimeSpan.FromSeconds(1));
    _consumer.Seek(new TopicPartitionOffset(topicPartition, watermarkOffsets.Low));
}

Which would work about 50% of the time, with the other half throwing an “erroneous” exception.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Spring Kafka - How to reset offset to latest with a group id?
Another way, we can always consume lastest message without commit group offset , by specify properties value with {"enable.auto.commit:false", " ...
Read more >
How to change or reset consumer offset in Kafka? - DIGIT Core
In this tutorial, we will go through the step by step process to reset the offset of the Kafka consumer group.
Read more >
How to reset consumer offsets in Apache Kafka
Next, you click the edit (pencil) icon next to the offset value in the affected consumer. Then you change the offset to the...
Read more >
Unable to reset offset for a single partition - Java Clients
offset.reset , I'm trying to explicitly reset the offset for each consumer, when it gets a state partition assigned. I tried calling ....
Read more >
Kafka Consumer Auto Offset Reset
reset set to latest, it does not consume the messages, and has no valid offset set. This is reflected in the CURRENT-OFFSET value...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found