Using DLQ in Spring Cloud Stream Binder Kafka does not commit offsets

Based on the Spring Cloud Stream Binder Kafka documentation:

autoCommitOnError Effective only if autoCommitOffset is set to true. If set to false it suppresses auto-commits for messages that result in errors, and will commit only for successful messages, allows a stream to automatically replay from the last successfully processed message, in case of persistent failures. If set to true, it will always auto-commit (if auto-commit is enabled). If not set (default), it effectively has the same value as enableDlq, auto-committing erroneous messages if they are sent to a DLQ, and not committing them otherwise. Default: not set.

I interpret this as, if my @StreamListener runs into an error, it will retry up to max-attempts then write to DLQ. Once the record is sent to DLQ, the offset should be committed. However, when I check the consumer group offset $ kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group my-consumer-group --describe, the current offset is not incremented.

This is my application configuration

spring:
  cloud:
    stream:
      default:
        group: my-consumer-group
      kafka:
        binder:
          brokers:
            - 127.0.0.1:9092
          producer-properties:
            retries: 2147483647
        bindings:
          input:
            consumer:
              auto-commit-offset: false
              autoCommitOnError: true # by default it it same as enableDlq
              enableDlq: true
              dlqName: test-dlq
              configuration:
                max.poll.interval.ms: 5000
      bindings:
        input:
          destination: test-topic
        output:
          destination: test-topic

My StreamListener is similar to below: I made an error case on purpose to see how DLQ and acks are handled.

@EnableBinding(SinkBinding.class)
public class KafkaConsumer {

    @StreamListener(value = SinkBinding.INPUT)
    public void listen(Message<String> message) throws InterruptedException {
        System.out.println("log this message : " + message.getPayload() + " offset : " + message.getHeaders().get(KafkaHeaders.OFFSET).toString());

        String a = null;
        a.substring(0, 3); // causing null pointer exception for testing purposes
        sendAck(message);
    }

    public void sendAck(Message<String> message) {
        Acknowledgment acknowledgment = message.getHeaders().get(KafkaHeaders.ACKNOWLEDGMENT, Acknowledgment.class);
        if (acknowledgment != null)  {
            try {
                acknowledgment.acknowledge();
                System.out.println("manual commit complete");
            }
            catch (Exception e)  {
                throw e; // retry
            }
        }
    }
}

Versions are spring-boot: 2.2.5.RELEASE spring-cloud: Hoxton.SR3

These are my test caes and observations

1. Start Kafka consumer app and send a record. Then it will try to process it three times and then fail. Then it is sent to DLQ. At this point, conusmer is idle (nothing happens). However, checking with kafka-consumer-group command, the offset is not increased.
2. Before starting the consumer app, send a few records. Then, start the consumer app. It will go through each record and send to DLQ. It should stop here like the first test case. However, once it iterates over all the records, it will go back to the starting offset and write it to DLQ again and again (it loops). When testing with 3 different records, I saw my DLQ was being loaded with those 3 records repeatedly.

Both caes would be handled properly if the offset was being committed correctly with DLQ.

Is this a desired behavior? To me it seems like a bug. Please let me know.

Thanks,

Jay

[garyrussell]

@sobychacko In the binder, the error handler that publishes to the DLQ needs to check if the container has a manual ack mode (MANUAL or MANUAL_IMMEDIATE) and, if so, use the Acknowledgment header to commit the offset after successfully publishing to the DLQ.

[sobychacko]

@jskim1991 As @garyrussell commented above, we need to fix this in the binder. Will do so before the next SR release.