Error messages are never sent to the DLQ

Hi,

I created a Spring Cloud Stream application using the Kafka Binder, here is my yml configuration file:

spring:
  cloud:
    stream:
      kafka:
        binder:
          brokers: localhost
          consumerProperties:
            schema.registry.url: http://localhost:8081
            key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
            value.deserializer: io.confluent.kafka.streams.serdes.avro.SpecificAvroDeserializer
          producerProperties:
            schema.registry.url: http://localhost:8081
            key.serializer: org.apache.kafka.common.serialization.StringSerializer
            value.serializer: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerializer
        bindings:
          input:
            consumer:
              startOffset: earliest
              enableDlq: true
              dlqName: dlq
      bindings:
        input:
          destination: foo
          group: fooGroup
          consumer:
            maxAttempts: 1
            useNativeDecoding: true
        output:
          destination: bar
          producer:
            useNativeEncoding: true
      function:
        definition: processor

Everything works great but the DLQ. While inspecting the broker I can see that it gets created yet no messages are sent to the topic. Here are the different processors I tried:

    @Bean
    public Function<EventA, EventB> processor() {
        return input -> {
            throw new RuntimeException("FAIL");
        };
    }

    @Bean
    public Function<Flux<EventA>, Flux<EventB>> processor() {
        return input -> input.map(i -> {
            throw new RuntimeException("FAIL");
        });
    } 

Libraries used:

    compile group: 'org.springframework.cloud', name: 'spring-cloud-stream', version: '2.2.0.RELEASE'
    compile group: 'org.springframework.cloud', name: 'spring-cloud-stream-binder-kafka', version: '2.2.0.RELEASE'
    compile group: 'org.springframework.cloud', name: 'spring-cloud-stream-schema', version: '2.2.0.RELEASE'

Did I miss something?

[sobychacko]

@antoin-m Apologies for the late response. I was able to triage your issue. It looks like you are missing some configuration. You are using native de-serialization (useNativeDecoding: true) and thus when you write the failed messages to DLQ, you need to use a corresponding serializer. In this case, you need to use io.confluent.kafka.streams.serdes.avro.SpecificAvroSerializer for DLQ. Here is an example of the configuration:

bindings:
  input:
     consumer:
       startOffset: earliest
       enableDlq: true
       dlqName: dlq
       dlqProducerProperties:
         configuration:
           key.serializer: org.apache.kafka.common.serialization.StringSerializer
           value.serializer: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerializer

If you are not using native decoding, then you don’t need to set configuration on dlqProducerProperties as it is handled by the framework using a converter.

I will use this issue to add some additional docs for this. Thank you!

[garyrussell]

Ah; sorry; I was referring to specific consumer/producer bindings, not the producer properties at the binder level.

Yes, I think this is a reasonable request, but it could be a breaking change so it will probably have to be done in the next release.