Question: Is maxUncommittedMessagesToHandle infinitely growable?

Hi there! Thank you for this great library, we are exploring this as an option for a cross-team use case whereby we cannot increase the number of partitions to scale consumption easily.

One question regarding a parameter, is maxUncommittedMessagesToHandle infinitely growable in terms of memory usage and overflow (I assume something like u64::MAX)? (or anything else)

The scenario we are planning for is that:

• We encounter a poison pill that is being infinitely retried
• We don’t have the resources to set up DLTs or retry topics
• We would still like to continue processing incoming messages in a timely manner, until manual intervention is done to move the committed offset past the poison pill, and restart the consumer.

If relevant we are thinking of using unordered or “ordered by key” processing modes.

[astubbs]

You’re most welcome! I really enjoy working on it. And there are some big improvements about to be released too (keep an eye only fork if curious).

Where did you see maxUncommittedMessagesToHandle, or are you speaking hypothetically? Ah! I just found it - oh dear. Yes, it’s in the readme, but the configuration was removed a long time ago, when we added per record ack persistence, didn’t think it was needed as it’s original intent was to prevent big replay upon rebalance (as per record ack wasn’t persisted). I’ll remove it from the docs.

At the moment, there is no limit enforced (although there used to be). If you would like this, please file a feature request. It hasn’t been something many have asked about, but I still think it’s a good idea.

So people should still monitor their traditional consumer lag as normal.

You can also monitor the record context in your user function, to see if a record has failed too many times, and decide what to do about it. So although I think it should be implemented, there are ways to handle the situation as is. But I’m keen to hear your thoughts!

[ang-zeyu]

Slightly unrelated but a little more about use case too:

Poison pills should be quite rare / non-existent. We currently have a RabbitMQ up and running with infinite retries and no DLQs for a couple years now, and am planning migration to Kafka. So DLT/retry topics are lower priority, but we’d still like to confirm what would happen in the event one occurs.

Our event volume is also fairly high (~< 10 million per day).