Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Question: Is maxUncommittedMessagesToHandle infinitely growable?

See original GitHub issue

Issue Description

Hi there! Thank you for this great library, we are exploring this as an option for a cross-team use case whereby we cannot increase the number of partitions to scale consumption easily.

One question regarding a parameter, is maxUncommittedMessagesToHandle infinitely growable in terms of memory usage and overflow (I assume something like u64::MAX)? (or anything else)

The scenario we are planning for is that:

  • We encounter a poison pill that is being infinitely retried
  • We don’t have the resources to set up DLTs or retry topics
  • We would still like to continue processing incoming messages in a timely manner, until manual intervention is done to move the committed offset past the poison pill, and restart the consumer.

If relevant we are thinking of using unordered or “ordered by key” processing modes.

Issue Analytics

  • State:open
  • Created 3 months ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

astubbscommented, Dec 8, 2022

You’re most welcome! I really enjoy working on it. And there are some big improvements about to be released too (keep an eye only fork if curious).

Where did you see maxUncommittedMessagesToHandle, or are you speaking hypothetically? Ah! I just found it - oh dear. Yes, it’s in the readme, but the configuration was removed a long time ago, when we added per record ack persistence, didn’t think it was needed as it’s original intent was to prevent big replay upon rebalance (as per record ack wasn’t persisted). I’ll remove it from the docs.

At the moment, there is no limit enforced (although there used to be). If you would like this, please file a feature request. It hasn’t been something many have asked about, but I still think it’s a good idea.

So people should still monitor their traditional consumer lag as normal.

You can also monitor the record context in your user function, to see if a record has failed too many times, and decide what to do about it. So although I think it should be implemented, there are ways to handle the situation as is. But I’m keen to hear your thoughts!

ang-zeyucommented, Dec 11, 2022

Slightly unrelated but a little more about use case too:

Poison pills should be quite rare / non-existent. We currently have a RabbitMQ up and running with infinite retries and no DLQs for a couple years now, and am planning migration to Kafka. So DLT/retry topics are lower priority, but we’d still like to confirm what would happen in the event one occurs.

Our event volume is also fairly high (~< 10 million per day).

Read more comments on GitHub >

github_iconTop Results From Across the Web

Maximum uncommitted messages (MAXUMSGS) - IBM
The number of allowed uncommitted messages should be greater than or equal to the sum of the message limit (MAXDEPTH) of all queues...
Read more >
View topic - max uncommitted message limit... -
I just use pub/sub simply, but my clients can not receive message after getting MQJMS2002 exception. This is a JMS error. What is...
Read more >
I have an uncommitted message on the queue – but no one ...
I have an uncommitted message on the queue – but no one has the queue open. Which application is causing the problem?
Read more >
message-oriented middleware – Page 2 – Brave New Geek
Messages written to the WAL are considered uncommitted or “dirty” initially. ... We can address this problem if we think about our logs...
Read more >
14.15 InnoDB Startup Options and System Variables
Specifies whether to record the pages cached in the InnoDB buffer pool when the MySQL server is shut down, to shorten the warmup...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found