Kafka processing prop throttled.unprocessed-record-max-age.ms unexpected behavior

Describe the bug

Hi guys! Recently I’ve noticed some interesting behavior in records from kafka processing: if I set throttled.unprocessed-record-max-age.ms = 12000, and then use Thread.sleep(8000) inside a record processing method like this:

@Blocking @Acknowledgment(value = POST_PROCESSING) @Incoming(“test-incoming”) public void receiveEvent(SomeObject event) { log.info(“starting processing event {}”, event.getId()); try { Thread.sleep(8000); } catch (Exception ex) {} log.info(“finished processing event {}”, event.getId()); }

And then I send, let’s say 10 messages into this topic. From time to time, it fails because of “hasTooManyMessagesWithoutAck” which is not kind of clear because the message is processed much faster than 12 seconds. I made a small investigation and found out that problem is not with the already processed message, but with the next one, which is waited for too long in Queue<OffsetReceivedAt> receivedOffsets (KafkaThrottledLatestProcessedCommit). This method takes part in hasTooManyMessagesWithoutAck consideration. The next message stays in receivedOffsets for some time before processing the record actually started. And it happens more often when @Retry annotation is used: 2 corrupted records (record processing ends with exception and takes all retries) in a row, and default throttled.unprocessed-record-max-age.ms = 60s is easily used in case of @Retry(delay = 5000, maxRetries = 6)

Expected behavior

throttled.unprocessed-record-max-age.ms config applies individually for each record after its processing starts. Otherwise, there should be a way to configure something like “processing.records.queue.max.size”, which does not equal 1 in the current implementation.

Actual behavior

throttled.unprocessed-record-max-age.ms config applies to all received offsets, even for records whose processing hasn’t started yet

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

java version “11.0.12” 2021-07-20 LTS

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.4.1.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 7.0

Additional information

I believe it’s not really a bug, just a way how throttled commit processing works. But I would be definitely glad to see a small explanation about such behavior in your commit strategies guide here

[ozangunalp]

I now understand the confusion. A record is considered “received” not when it is polled from the broker but depending on how the downstream processing “requests” records to consume.

In most cases, the processing is sequential and the downstream processor make one request each time it acks/nacks a record.

However, having the first two requests at the same time seems odd. I need to take a look. I suspect that it has to do with the part of the throttled commit strategy which initializes the offset store by fetching committed offsets from the broker for a topic partition.

[cescoffier]

Closing - was a question (use discussion instead of issues).