Consumer.seek commits an offset, causing possible data and metadata loss

In debugging one of our stream processing tasks we noticed that consumer.seek commits an offset in order to affect the consumer’s fetching position. This mixes controlling the consumer position with recording progress and can lead to offset-metadata getting lost.

Offset metadata can be very useful when working with replicating to and from local stores, especially for achieving exactly-once processing, and is used by KafkaStreams for exactly this purpose (emphasis mine):

• When committing a task, after the state stores have all been flushed and before produce write the committed offset, we can choose to trigger the persistent state stores’ checkpoint mechanism to make a checkpoint for each persistent stores.
• When writing the committed offset into the input-offset-topic, also pass in the reference (e.g. local state directory / files locations) of all the persistent store checkpoints into the offset metadata as part of the state triplet.
• Then upon resuming, if the fetched last transaction contains the state store checkpoints information, we can skip the restoration step of the state stores from changelog but use the checkpointed image directly.

When seek commits an offset, this metadata is subsequently lost.

Another example is where you produce to the same topic that you’re consuming (eating your own tail), which is useful for things like implementing uniqueness constraints (described well here). There it can often be useful to seek back 1 from the last committed offset, to kickstart the process (as if you don’t, no messages are ever produced for you to continue from), especially when doing things like replicating from a store to a Kafka topic.

As far as I can tell, the Java KafkaConsumer does not commit the offset, but only changes the offset in memory so the next fetch happens from there.

Proposed solution

Eventually, I think we’d want to be in a situation where:

• consumer.seek only affects what message is fetched next, not how progress has been made thus far.
• consumer.commit can be used to record progress, possible after a seek, if that’s required. See #378.

However, this would constitute a breaking change, despite the committing of the offset is currently a bit of a side effect. So perhaps a flag passed to consumer.seek would be a way to opt-in to the new behaviour, as a way to verify the approach in real applications. That way if the API is to be broken in a new major version, we prevent the need for a revert.

[Nevon]

@Nevon would the kafkajs project take sponsorships for this development work?

Thank you very much for offering! I’m personally not set up to take on contract work, but I’ll do you one better and put up a free PR . #1012

If you want to support my work, consider becoming a sponsor.

[Nevon]

The fix for this will be out in 1.16.0-beta.7 in a few minutes. Let me know if it solves your problem, @angelozerr.