Repeated messages of shared dispatcher

Describe the bug Recently I found the C++ tests BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub and BasicEndToEndTest.testpatternMultiTopicsHttpConsumerPubSub became flaky. Eventually I found it’s caused by the duplicated messages.

To Reproduce

First, start a standalone with latest master.

Then, run the following Java test.

    @Test(invocationCount = 10)
    public void test() throws Exception {
        @Cleanup final PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl("http://127.0.0.1:8080").build();
        final String topicBase = "persistent://public/default/my-topic-" + System.currentTimeMillis();
        for (int i = 0; i < 3; i++) {
            admin.topics().createPartitionedTopic(topicBase + "-" + i, i + 2);
        }
        @Cleanup final PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
        final Consumer<byte[]> consumer = client.newConsumer()
                .topicsPattern(topicBase + ".*")
                .subscriptionName("sub")
                .receiverQueueSize(10)
                .subscriptionType(SubscriptionType.Shared)
                .subscribe();
        final int numMessages = 100;
        for (int i = 0; i < 3; i++) {
            final Producer<byte[]> producer = client.newProducer()
                    .topic(topicBase + "-" + i)
                    .messageRoutingMode(MessageRoutingMode.RoundRobinPartition)
                    .create();
            for (int j = 0; j < numMessages; j++) {
                producer.send(("msg-content-" + i + "-" + j).getBytes());
            }
        }
        int numReceived = 0;
        while (true) {
            final Message<byte[]> msg = consumer.receive(1, TimeUnit.SECONDS);
            if (msg == null) {
                break;
            }
            numReceived++;
        }
        Assert.assertEquals(numReceived, numMessages * 3);
    }

Sometimes it failed. (You can add more logs or debugged code to see the received messages of all partitions)

Expected behavior The tests should always succeed.

Screenshots The failure in my local env.

Additional context Adding some logs into C++ tests could make tests more stable to pass. And the Java test looks less flaky than the C++ tests.

After applying dispatcherDispatchMessagesInSubscriptionThread=false in conf/standalone.conf, the Java test could pass even if invocationCount = 50. Changing the subscription type to Exclusive also works.

From the test result, we can see the bug was introduced from https://github.com/apache/pulsar/pull/16603 that we should not run sendMessagesToConsumers in another thread.

[eolivelli]

in #16603 I had to change a test that was failing in a similar way. but the test was missing a “acknowledge(message)” so I thought that the test was wrong.

https://github.com/apache/pulsar/pull/16603/files#diff-33303e92b2d03115c1bde9ad3af39f9127f6a25270855fb5b5c2eb7c3db74e50R281

I will revert the change to the test and add your new test case

[tisonkun]

It seems the exception is timeout instead of unexpected result, reported at https://github.com/apache/pulsar/issues/17008.