Handle when the consumer `.eachMessage` failed `rertries`

I want to implement my own DLQ(Dead Letter Queue). When retries=5, the consumer retry .eachMessage function 5 times and restart if the message handler throws the error.

I want to implement my own handler by catching NumberOfRetriesExceed event. My own handler will have a logic that produces a message to a specific topic and commits the offset instead of restarting the Kafka client.

How can I catch the event?

[Nevon]

when eachMessage is called with a message, can we assume there are no network errors or partial data up until that point?

You cannot get partial data in your callbacks. However, your question is unrelated to this thread. Github Issues is not the place for support. Try StackOverflow or the Slack community in the future.

[Nevon]

I understand where the confusion is coming from. KafkaJS does not retry any error that is thrown from within your message processing - only errors that happen internally to KafkaJS. If your eachMessage throws because you have some issue inside it, that does not get retried by KafkaJS.

You can try it yourself quite easily:

await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
        throw new Error('💣')
    },
})

If you run that, you’ll see an error message saying: Error when calling eachMessage, along with the topic, partition and offset, but it won’t do any retries. It will just crash and restart the consumer.

If you want to retry your own message processing before giving up and sending the message to your DLQ, you’ll have to build that yourself within your eachMessage handler. We don’t export the retry functionality, so you’ll have to do it yourself though. This seems like a reasonable approach if you don’t want to roll your own.