What is expected behavior when combining 100GB retention policy, 1GB backlog quota and a consumer with SubscriptionInitialPosition.Earliest setting?
See original GitHub issueI couldn’t get clear understanding of pulsar behavior from documentation in the following situation.
Supposing a namespace configured 100GB retention policy, 1GB backlog quotas and 100GB messages are pushed with no subscription, what will happen if a subscription with SubscriptionInitialPosition.Earliest
setting is created?
Is the backlog of the newly created subscription over 1GB backlog quotas?
Will a producer immediately stop because of producer_request_hold
policy?
I originally discussed about this question on slack and was recommended to rise issue in order to get clarification. https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1594348193122700
Issue Analytics
- State:
- Created 3 years ago
- Comments:16 (5 by maintainers)
Top Results From Across the Web
Message retention and expiry - Apache Pulsar
The unacknowledged messages are controlled by the backlog quota. When a retention limit on a topic is exceeded, the oldest message is marked...
Read more >TGI Pulsar - episode 016: backlog vs storage size - SlideShare
... the expected behavior when combining 100GB retention policy, 1GB backlog quota and a consumer with SubscriptionInitialPosition.Earliest ...
Read more >Understanding Pulsar Message TTL, Backlog, and Retention
Rule #1: Pulsar keeps all messages until they are acknowledged. Pulsar's subscription keeps track of all message consumption. When a consumer ...
Read more >Apache Pulsar | Fusion 5.4 - Lucidworks Documentation
Backlog quota size: 1GB. Backlog retention policy: consumer_backlog_eviction is turned on. How do I change heap memory settings?
Read more >Frequently Asked Questions
Subscriptions can be dropped by explicitly unsubscribing (in Consumer API) ... In Pulsar, “namespaces” are the administrative unit: you can configure most ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@sijie Thank you for your answer. That is to say, in order for lately created subscription to read all retained messages without interfering a producer, should backlog quota be set to the same size as retention policy? (In this case, 100GB.) If so, as retention policy is applied to the messages only not in backlog, do I need to estimate the maximum size of disk space at twice the size of the retention policy?
I see. I’m also interested in how retention policy and backlog quotas affect compactioned topic. I think there is not much information on this.