Kafka messages are being lost on INVALID_FETCH_SESSION_EPOCH

Recently we have observed that whenever there is an issue with INVALID_FETCH_SESSION_EPOCH from Kafka client the corresponding message is being skipped and hence it’s lost in transit. INVALID_FETCH_SESSION_EPOCH is a known issue that was fixed in Kafka client 2.3.0 + according to KAFKA-8052. However, what we observed is when this is printed in logs the corresponding message is skipped entirely. This is impacting the at-least-once guaranteed message delivery part of Spring Cloud Stream, so I am not sure if this is a known issue or not. Obviously, with the most recent versions of Spring Cloud Stream, this issue should not happen but it raises a point that under similar circumstances a message could be skipped due. I just want to raise this issue here due to the importance of losing messages in transit.

"@timestamp":"2021-08-11T13:40:44.081+00:00","message":"[Consumer clientId=consumer-9, groupId=tag-modifiers] Node 17 was unable to process the fetch request with (sessionId=1995608629, epoch=331): INVALID_FETCH_SESSION_EPOCH.","logger_name":"org.apache.kafka.clients.FetchSessionHandler","thread_name":"KafkaConsumerDestination{consumerDestinationName='edit-tag', partitions=9, dlqName='dlq'}.container-0-C-1","level":"INFO","profile":"beats","microservice":"data-service"

This issue is observed for Spring Cloud Stream 2.2.0 which uses Kafka client 2.0.1, but it should happen for any version of Spring Cloud Stream that uses Kafka client before 2.3.0.

I was under the impression that an exception in Kafka client to fetch a message should trigger the consumer to retry to fetch a message, but it seems that’s not what’s happening.

[garyrussell]

I have no idea - ask the Kafka folks - as I said, it’s totally out of our control.

[sobychacko]

@mraliagha Closing the issue. Please feel free to re-open with more context if you think that is necessary.