Cannot parse <null> schema

Describe the bug

Reconnecting an updated consumer application that had NOT changed any of the following:

• Any connection or topic details
• Any message object details used to generate the AvroSchema

This system was restarted from an update of application code unrelated to pulsar handling, and has two consumers. At first both consumers were failing but after a restart one is OK and the other continues to fail with the error:

org.apache.pulsar.shade.org.apache.avro.SchemaParseException: Cannot parse <null> schema

I could see the schemas using pulsar-admin, and they look exactly as they should, and the producers are tied to the same.

For the two consumar connections in this application they should have been binding to schema versions 1 & 2 respectively, but both had the error mentioning schema version 0

My connection code:

consumerIngressV1 = pulsarClient.newConsumer(AvroSchema.of(IngressMessageV1.class))
    .subscriptionName(subscriptionName)
    .topic(topics)
    .subscribe();

The stacktrace:

2020-10-27 22:48:12,322 [pool-3-thread-8] INFO o.a.p.client.impl.schema.AvroSchema - Load schema reader for version(0), schema is : , schemaInfo: {
 "name": "",
 "schema": "",
 "type": "NONE",
 "properties": {}
} -- 
2020-10-27 22:48:12,323 [pool-3-thread-8] ERROR io.pethau.pl.PipelineService - Exception: org.apache.pulsar.shade.org.apache.avro.SchemaParseException: Cannot parse <null> schema - {} - {} - {} -- 
org.apache.pulsar.shade.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.pulsar.shade.org.apache.avro.SchemaParseException: Cannot parse <null> schema
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3973)
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4957)
    at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:107)
    at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:293)
    at io.pethau.pl.PipelineService.lambda$run$2(PipelineService.java:103)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.pulsar.shade.org.apache.avro.SchemaParseException: Cannot parse <null> schema
    at org.apache.pulsar.shade.org.apache.avro.Schema.parse(Schema.java:1597)
    at org.apache.pulsar.shade.org.apache.avro.Schema$Parser.parse(Schema.java:1396)
    at org.apache.pulsar.shade.org.apache.avro.Schema$Parser.parse(Schema.java:1384)
    at org.apache.pulsar.client.impl.schema.StructSchema.parseAvroSchema(StructSchema.java:182)
    at org.apache.pulsar.client.impl.schema.AvroSchema.loadReader(AvroSchema.java:99)
    at org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:75)
    at org.apache.pulsar.client.impl.schema.StructSchema$1.load(StructSchema.java:72)
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2276)
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
    at org.apache.pulsar.shade.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
    ... 11 common frames omitted

To Reproduce Unable to re-produce.

Expected behaviour Consumers connect to the topics using the latest version from before the restart.

Desktop (please complete the following information):

• OS: Docker, standalone, v2.6.0

Additional context None.

[congbobo184]

@robshep @dhinesherode91 could you please check if the topic has multiple schemas by the problem when it happens? or upgrade the pulsar version and test if this problem has been fixed

@tisonkun it seems like a user uses problem.

[dhinesherode91]

Thanks for the information.

In my case, in my local I am running a standalone Pulsar and Consumer applications are Spring boot application. The above issue is not occurring in local.

But in staging and production setups, we are running Pulsar in kubernetes and our Consumer spring boot applications are running as Containers. In those setups I could see this issue more often and I have to restart my containers to make it work.

On debugging we could see like the schema is not available in the consumer application pulsar client local cache and due to that on every message receive when the pulsar client tries to convert that respective message into respective class object by parsing the schema, it throws this ‘Cannot parse null schema’ error.

Something related to having/loading SchemaInfo in the memory. It would be great if someone from the team could help us in fixing it.