Header lost in DLQ failure strategy

Describe the bug

I am implementing a retry policy in my project in quarkus. Here’s the pitch : I produce a Kafka message who’s consumed in several place in my project. If for any reason one of this consumer fails to handle this message, it’s nacked and publish in a retry topic using the dead-letter-queue failure strategy.

The messages from this topic are handle by another consumer who try to republish it in the original topic while adding a “retryCount” header. The message will then be consumed again by the faulty consumer, who will either succeed or fail again and publish it back in the retry topic.

Here is my issue :

The headers won’t persist after first retry. After first fail, the message is succesfully pushed in the error topic, then pushed back in its original topic with the new custom header, but if the consumer fail once more and push the message in the error topic again, my custom header doesn’t exist anymore.

https://github.com/smallrye/smallrye-reactive-messaging/blob/d09d8b97cba72988f7c06f6ceec81b4a1a322a7d/smallrye-reactive-messaging-kafka/src/main/java/io/smallrye/reactive/messaging/kafka/fault/KafkaDeadLetterQueue.java#L129 This seems to show that headers are kept when pushed in DLQ but in my consumer, it doesn’t

@ApplicationScoped
@ActivateRequestContext
@Slf4j
public class KafkaErrorConsumer {
private final KafkaRetry kafkaRetry;
private String insanity = "Insanity is doing the same thing over and over again and expecting a different result";

@Inject
public KafkaErrorConsumer(KafkaRetry kafkaRetry) {
    this‧kafkaRetry = kafkaRetry;
}

@NonBlocking
@Incoming(KafkaTopics.KAFKA_ERROR_EVENT)

public CompletionStage<Void> handle(Message<String> event) throws BusinessException, InterruptedException {

var metadata = event‧getMetadata(IncomingKafkaRecordMetadata‧class)
                    .orElseThrow(() -> new IllegalArgumentException("Expected a message coming from Kafka"));
String reason = new String(metadata‧getHeaders().lastHeader("dead-letter-reason").value());
String topic = new String(metadata‧getHeaders().lastHeader("dead-letter-topic").value());

Instant timeStamp = metadata‧getTimestamp();
if (!reason‧equals(insanity)) {
    log‧info("""
                                
            Received failed event : {}
            from : {}
            because of : {}""", event‧getPayload(), topic, reason);
    kafkaRetry‧execute(event, topic);

    return event‧ack();
} else {
    metadata‧getHeaders().forEach(header -> log‧info(new String(header‧value())));

I enter the “else” durind second retry, so the custom header shoud be there, as it was added after first fail, but it’s not.

Expected behavior

headers should be kept while pushing the message in the DLQ, as shown in this : https://github.com/smallrye/smallrye-reactive-messaging/blob/d09d8b97cba72988f7c06f6ceec81b4a1a322a7d/smallrye-reactive-messaging-kafka/src/main/java/io/smallrye/reactive/messaging/kafka/fault/KafkaDeadLetterQueue.java#L129

Actual behavior

Headers are overwritten and metadata‧getHeaders().lastHeader("retryCount") return null

How to Reproduce?

How to rerpoduce :

1. nack a message in a topic with dead letter queue failure strategy enabled via quarkus conf
2. consume this message, add a custom header, push it in its original topic
3. consume this message again, check the custom header, nack the message if the header is present so it goes in the DLQ topic
4. consume this message once more, custom header is gone

Output of uname -a or ver

Linux 5.14.7-2-MANJARO #1 SMP PREEMPT Wed Sep 22 12:22:56 UTC 2021 x86_64 GNU/Linux

Output of java -version

openjdk version “17” 2021-09-14 OpenJDK Runtime Environment (build 17+35-2724) OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.2.3.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.2 (NON_CANONICAL)

Additional information

No response

[sergejomon]

Ok ! So that would make sens : header are kept from C to B but not from B to C. I’ll try to update to quarkus 2 and last version of smallrye in service B and I’ll update this issue accordingly. Thank you for your time 🙂

---

De : Clement Escoffier @.> Envoyé : lundi 6 décembre 2021 08:19 À : quarkusio/quarkus @.> Cc : Serge @.>; Author @.> Objet : Re: [quarkusio/quarkus] Header lost in DLQ failure strategy (#20537)

The in-memory is just for tests, so we can ignore it. I believe that Service B, which uses an “old” version of Quarkus may not have the feature we are discussing here. A and C should have it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/quarkusio/quarkus/issues/20537#issuecomment-986503295, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJLB2ZUUHI6DKZLJTZXMPFTUPRPXDANCNFSM5FLLEMDQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[cescoffier]

The in-memory is just for tests, so we can ignore it. I believe that Service B, which uses an “old” version of Quarkus may not have the feature we are discussing here. A and C should have it.