Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Shading Avro in pulsar-client
See original GitHub issueDescribe the bug
Because we shade Avro in the pulsar-client, the ReflectData API to get a schema from a POJO fails to get the correct schema for generated Avro java classes
To Reproduce
Given an avro schema:
{
"namespace": "example.avro",
"type": "record",
"name": "User",
"fields": [
{
"name": "name",
"type": "string"
},
{
"name": "favorite_number",
"type": [
"int",
"null"
]
},
{
"name": "favorite_color",
"type": "string"
},
{
"name": "age",
"type": "int",
"default": 19
}
]
}
Passing the generated java class from the above schema to the AvroSchema in pulsar-client produces the following result:
System.out.println(new String(org.apache.pulsar.client.impl.schema.AvroSchema.of(User.class).getSchemaInfo().getSchema()));
outputs:
{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":["null","string"],"default":null},{"name":"favorite_number","type":["null","int"],"default":null},{"name":"favorite_color","type":["null","string"],"default":null},{"name":"age","type":"int"}]}
which is incorrect while
System.out.println(ReflectData.get().getSchema(User.class));
outputs the correct schema that matches the original schema:
{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":"string"},{"name":"age","type":"int","default":19}]}
Here is the line in Avro responsible for getting the schema from generated classes: https://github.com/apache/avro/blob/release-1.8.2/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificData.java#L277
Here is a project I created that demonstrates this problem:
Issue Analytics
- State:
- Created 5 years ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
@vzhikserg Thanks, that will work for me, though an underlying Avro upgrade probably would have been a good time to uncouple the Avro version from Pulsar to avoid this same problem in the future…
I’d like to at least have the option to use the Java Pulsar client with an unshaded Avro. We use Avro extensively with Kafka, and while we migrate to Pulsar, its imperative we have compatibility at the Avro message level, which means using the same version of Avro.
Here is an issue we are running into because our messages use Avro 1.9.1 which supports the java.time APIs: