Unable to consume messages, need to re-subscribe client

This is similar to issue 2013. We have several shared subscriptions of partitioned topics and we found some cases where a subscription’s client is not receiving messages, but it’s not related to a specific topic partition. In this case, if we re-subscribe the client (PulsarClient.subscribeAsync), messages start to be delivered as expected.

Client logs this warn:

[log_time:16:17:42.377] [thread:pulsar-timer-6-1] [level:WARN ] [logger:UnAckedMessageTracker] - [ConsumerBase{subscription='picking-consumer-prod.fbm-wms-picking', consumerName='i-08e97d991d75e9d07-10.73.125.156', topic='persistent://fury/global/wms-wave-feed-prod.fbm-wms-picking/wms-wave-feed-prod.fbm-wms-picking'}] 1 messages have timed-out

And we don’t see any error from brokers.

Any idea? Any log or metric that may be useful to debug this problem?

System configuration

Pulsar version (brokers): 1.20.0 Pulsar clients version: 1.22.0 Bookkeeper version: 4.7.1

[Poluect]

Hello! We started using Pulsar instead of Kafka in our system, and currently facing the same issue, when few subscriptions (like 1 - 3 of 30) stop receiving messages, and they get stuck in a backlog, until either a subscriber, or pulsar proxy restarted.

We are using version 2.4.2, with 3 brokers in a cluster. Issue appears pretty often, like 3 - 6 times in a day. Stats also attached. Hope this issue can be fixed or at least hope there can be any workaround except of force deleting kube pods with subscriptions to make them reconnect. We used Kafka to handle some critical parts of our system, unfortunately we can’t rely on Pulsar without having simplest consumer - producer scenario work well. Thank you!

topic-stats-internal.txt topic-stats.txt

[sijie]

@Poluect thank you for your update! I would close this issue for now. Please re-open one if there is still an issue in the latest release.