Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Question: Is maxUncommittedMessagesToHandle infinitely growable?

See original GitHub issue

Issue Description

Hi there! Thank you for this great library, we are exploring this as an option for a cross-team use case whereby we cannot increase the number of partitions to scale consumption easily.

One question regarding a parameter, is maxUncommittedMessagesToHandle infinitely growable in terms of memory usage and overflow (I assume something like u64::MAX)? (or anything else)

The scenario we are planning for is that:

We encounter a poison pill that is being infinitely retried
We don’t have the resources to set up DLTs or retry topics
We would still like to continue processing incoming messages in a timely manner, until manual intervention is done to move the committed offset past the poison pill, and restart the consumer.

If relevant we are thinking of using unordered or “ordered by key” processing modes.

Issue Analytics

State:
Created 3 months ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

astubbscommented, Dec 8, 2022

You’re most welcome! I really enjoy working on it. And there are some big improvements about to be released too (keep an eye only fork if curious).

Where did you see maxUncommittedMessagesToHandle, or are you speaking hypothetically? Ah! I just found it - oh dear. Yes, it’s in the readme, but the configuration was removed a long time ago, when we added per record ack persistence, didn’t think it was needed as it’s original intent was to prevent big replay upon rebalance (as per record ack wasn’t persisted). I’ll remove it from the docs.

At the moment, there is no limit enforced (although there used to be). If you would like this, please file a feature request. It hasn’t been something many have asked about, but I still think it’s a good idea.

So people should still monitor their traditional consumer lag as normal.

You can also monitor the record context in your user function, to see if a record has failed too many times, and decide what to do about it. So although I think it should be implemented, there are ways to handle the situation as is. But I’m keen to hear your thoughts!

0reactions

ang-zeyucommented, Dec 11, 2022

Slightly unrelated but a little more about use case too:

Poison pills should be quite rare / non-existent. We currently have a RabbitMQ up and running with infinite retries and no DLQs for a couple years now, and am planning migration to Kafka. So DLT/retry topics are lower priority, but we’d still like to confirm what would happen in the event one occurs.

Our event volume is also fairly high (~< 10 million per day).