[BUG] KafkaConsumer.position() Method should give the offset one larger than the last record

Describe the bug According to Kafka Conusmer API Doc: https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

The position of the consumer gives the offset of the next record that will be given out. It will be one larger than the highest offset the consumer has seen in that partition. It automatically advances every time the consumer receives messages in a call to poll(Duration).

While currently, KoP will return the offset of the last record, without adding one to it

To Reproduce Steps to reproduce the behavior:

1. Suscribe to a Kafka topic or a KoP topic
2. Call conusmer.seekToEnd() and then print conusmer.position() result
3. Kafka will return the offset one larger than last record, while KoP not.

Expected behavior KafkaConsumer.position() Method should give the offset one larger than the last record

[BewareMyPower]

Got it. I’ll try to fix this issue after KoP 2.7.0 is released.

---

and I’am a little comfused why we need to request.offset - 1 when fetch.

When Kafka handles fetch requests, it reads data from request.offset.

However when KoP handles fetch requests, the workflow is:

1. Convert request.offset to a message id;
2. Read data after the message id.

Kafka consumer maintains an internal offset to construct FETCH request. The offset will be updated after resetOffsetsAsync. Currently KoP returns a latestOffset not latestOffset + 1 if consumer resets offset to latest. If we change it to latestOffset + 1, the consumer may fetch after latestOffset + 1.

[WalterMa]

As for how to fix this issure, please allow me to make a few comments. I’am not familiar with this project, so if any wrong, please correct me.

Can we add one to lastOffset only for the postsion() request ?

when consumer send FETCH requests, KoP did like fetchFromOffset(request.offset - 1)

, and I’am a little comfused why we need to request.offset - 1 when fetch.