question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

What is expected behavior when combining 100GB retention policy, 1GB backlog quota and a consumer with SubscriptionInitialPosition.Earliest setting?

See original GitHub issue

I couldn’t get clear understanding of pulsar behavior from documentation in the following situation.

Supposing a namespace configured 100GB retention policy, 1GB backlog quotas and 100GB messages are pushed with no subscription, what will happen if a subscription with SubscriptionInitialPosition.Earliest setting is created? Is the backlog of the newly created subscription over 1GB backlog quotas? Will a producer immediately stop because of producer_request_hold policy?

I originally discussed about this question on slack and was recommended to rise issue in order to get clarification. https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1594348193122700

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:16 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
hozumicommented, Jul 14, 2020

@sijie Thank you for your answer. That is to say, in order for lately created subscription to read all retained messages without interfering a producer, should backlog quota be set to the same size as retention policy? (In this case, 100GB.) If so, as retention policy is applied to the messages only not in backlog, do I need to estimate the maximum size of disk space at twice the size of the retention policy?

0reactions
hozumicommented, Jul 31, 2020

I see. I’m also interested in how retention policy and backlog quotas affect compactioned topic. I think there is not much information on this.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Message retention and expiry - Apache Pulsar
The unacknowledged messages are controlled by the backlog quota. When a retention limit on a topic is exceeded, the oldest message is marked...
Read more >
TGI Pulsar - episode 016: backlog vs storage size - SlideShare
... the expected behavior when combining 100GB retention policy, 1GB backlog quota and a consumer with SubscriptionInitialPosition.Earliest ...
Read more >
Understanding Pulsar Message TTL, Backlog, and Retention
Rule #1: Pulsar keeps all messages until they are acknowledged. Pulsar's subscription keeps track of all message consumption. When a consumer ...
Read more >
Apache Pulsar | Fusion 5.4 - Lucidworks Documentation
Backlog quota size: 1GB. Backlog retention policy: consumer_backlog_eviction is turned on. How do I change heap memory settings?
Read more >
Frequently Asked Questions
Subscriptions can be dropped by explicitly unsubscribing (in Consumer API) ... In Pulsar, “namespaces” are the administrative unit: you can configure most ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found