Schema prevents usage of Dead Letter Queue DLQ

Describe the bug When using a Schema pushing a negativeAcknowledged Message to a DeadletterQueue will fail. There is no obvious Errors/Exceptions. The Message, just stay in the current Topic and is never published to the DLQ-Topic. When using no Schema oder Schema.STRING it will work.

This happende when using pulsar 2.7.1 and the Java-Client.

Pulsar was started as standalone docker container.

$ docker run -it \
  -p 6650:6650 \
  -p 8080:8080 \
  --mount source=pulsardata,target=/pulsar/data \
  --mount source=pulsarconf,target=/pulsar/conf \
  apachepulsar/pulsar:2.7.1 \
  bin/pulsar standalone

To Reproduce I created an repo with some code for this issue: https://github.com/jvoigt/pulsar-schema-dlq-issue

Steps to reproduce the behavior:

1. Create Producer to topic with Schema pulsarClient.newProducer(JSONSchema.of(Payload.class)).create()
2. Send Message to Producer producer.send( new Payload("My Json message"))
3. Consume Topic with Schema AND DLQ pulsarClient.newConsumer(schema).topic(topic) .deadLetterPolicy(DeadLetterPolicy.builder().maxRedeliverCount(3).build()).subscribe()
4. receive the Message, but use a nack. consumer.negativeAcknowledge(msg);
5. The Message will be redeliverd 3 Times, then the DLQ-Topic ist automatically create.
6. But the Message is never sent.

Logs See the attached Repo for detailed Logs: But Mainly you can see that the client is creating a new Producer for the DLQ-Topic and then nothing more happens.

Apr 26, 2021 8:19:13 AM com.jvoigt.issue.MyListener received
INFO:  persistent://public/default/jsonTopic => Message received: {"message":"My Json message"}
Apr 26, 2021 8:19:13 AM com.jvoigt.issue.MyListener received
SEVERE: persistent://public/default/jsonTopic => I WILL NACK IT with Counter: 3
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.UnAckedMessageTracker$2 run
WARNING: [ConsumerBase{subscription='subscription', consumerName='5416a', topic='jsonTopic'}] 1 messages have timed-out
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Starting Pulsar producer perf with config: {
  "topicName" : "jsonTopic-subscription-DLQ",
.....
}
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Pulsar client config: {
  "serviceUrl" : "pulsar://localhost:6650",
.....
}
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerImpl connectionOpened
INFO: [jsonTopic-subscription-DLQ] [null] Creating producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerImpl lambda$connectionOpened$16
INFO: [jsonTopic-subscription-DLQ] [standalone-6-1] Created producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:20:04 AM org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl lambda$init$0
INFO: [jsonTopic] [subscription] [5416a] Prefetched messages: 0 --- Consume throughput received: 0.07 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 0.00 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Starting Pulsar producer perf with config: {
  "topicName" : "jsonTopic-subscription-DLQ",
.....
}
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Pulsar client config: {
  "serviceUrl" : "pulsar://localhost:6650",
.....
}
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerImpl connectionOpened
INFO: [jsonTopic-subscription-DLQ] [null] Creating producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerImpl lambda$connectionOpened$16
INFO: [jsonTopic-subscription-DLQ] [standalone-6-2] Created producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]`

thank you all in advance 😃

[linlinnn]

@jvoigt I work well locally after add default constructor, could you share your code?

[codelipenghui]

The issue had no activity for 30 days, mark with Stale label.