[bug](indeviduallyDeleteMessage): slow rate consumer but ack in order will get indeviduallyDeleteMessage

Describe the bug In my current test found, if i create a producer to produce with no dispatch limit while create a consumer to consumer with slow rate (set a 1 second sleep in message receive), although the the consumer acks in order but there is indeviduallyDeleteMessage in the topic.

And the weird thing is if i remove the 1 second sleep in consumer or i move the 1 second sleep from consumer to producer, there is no indeviduallyDeleteMessage in the topics. It looks like when the consumer is much slower than the producer, this issue will come.

This is the consume code i used, i add a delay for 1 second but the ack should be in order.

for true {
		msg, err := consumer.Receive(context.Background())
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
			msg.ID(), string(msg.Payload()))
		consumer.Ack(msg)

		time.Sleep(time.Duration(1) * time.Second)
	}

To Reproduce Steps to reproduce the behavior:

1. Start the producer without dispatch limit
2. Start the consumer with 1 second limit to consume
3. Stop the consumer after several consumes and acks are sent.
4. Check the topic stats-internal and there is some indeviduallyDeleteMessage.

Expected behavior The ack is in order, there is no indeviduallyDeleteMessage in the topic.

Screenshots I use golang-client to produce and consume

keep on producing and stop the consumer after several consume with acks

Check the topic stats-internal and there is some indeviduallyDeleteMessage.

The test code i used pulsar-client-go-demo.zip

Additional context golang client and java client both tested with this issue.

[gaozhangmin]

I would like to help on it.

[hangc0276]

This is the expected behavior. It was introduced by https://github.com/apache/pulsar/wiki/PIP-33%3A-Replicated-subscriptions

In order to support replicated subscription stats between different pulsar clusters, it will start subscription snapshot every 1 second by default if we enable this feature in consumer subscription replicateSubscriptionState(true).

The snapshot will write an internal mark into the topic message and will be filter out on message dispatch period. Thus you will see the messageId has been acked even though you haven’t ack the message yet. In fact, you will never receive this message.