Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Preparing for Protobuf and JSON Schema support

See original GitHub issue

The confluent schema registry as of version 5.5 supports 2 new schema types, Protobuf and Json Schema, see this article. In order to ultimately support these new schema types I would like to make some proposals to the users/contributors of this library.

Goals Separate specifics of the Avro schema/serdes from internals of the library. (Cache, Wire Encoder/Decoder, Subject generation) Maintain backwards compatibility with older Schema Registry deployments.

Separate Avro from internals

Cache - change the cache to just store the schema string, since other schema types may not be in Json Wire Encoder/Decoder - change the encoder/decoder to just do the wire encoding/decoding Subject Generation - delegate subject generation to specific schema serdes, and future iteration implement the different strategies

All of these changes should push the library to create a Serdes interface that might look something like this.

interface Serdes {
  serialize(schema: ConfluentSchema, payload: any) : Buffer
  deserialize(schema: ConfluentSchema, buffer: Buffer) : any 

Backwards Compatibility

In version 5.5.0 the schema registry introduces a new field to the various apis called schemaType. Reference here. Since the default is AVRO, and older versions fail when using this field, if the client uses AVRO schema/serdes it should omit the field.

There is a potential for breaking Api changes in this refactor, but users should not need to upgrade their Schema Registry deployment.

Note: I have a rough draft branch of some of these ideas, and want to get feedback before moving forward

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:5
  • Comments:10 (7 by maintainers)

github_iconTop GitHub Comments

Malkizcommented, Feb 1, 2021

Hi @dskatz22 and @Nevon , What is the status of this issue?

@dskatz22 I have read the code changes in your branch, and I think I understand it. I would be happy to continue the work you did to add support for Protobuf and JSON Schema.

I just wanted to make sure that this direction is considered acceptable by the library authors/maintainers ( @erikengervall ? ).


Nevoncommented, Feb 1, 2021

While I’m not the primary author of this library, and am hesitant to take on responsibility for it (hello @erikengervall 👋), I’d be happy to take a look at a pull request and provide feedback.

Curse my previous self!

I’m currently swamped with being the primary maintainer of KafkaJS along with my day job, so I don’t feel like I have the bandwidth to also actively maintain @kafkajs/confluent-schema-registry and cannot commit to helping driving this change through.

In terms of general thoughts, with JSON Schema and Protobuf being officially supported in Confluent schema registry, it makes perfect sense that we should also do so. I can’t speak to @dskatz22’s implementation, as I don’t have the necessary background on how this is implemented on the registry side and how other clients are working with it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Confluent Platform Now Supports Protobuf and JSON Schema
The new Protobuf and JSON Schema formats are supported in Schema Registry in addition to all other components of Confluent Platform 5.5.
Read more >
chrusty/protoc-gen-jsonschema: Protobuf to JSON-Schema ...
Protobuf to JSON-Schema compiler ... This takes protobuf definitions and converts them into JSONSchemas, which can be used to dynamically validate JSON messages....
Read more >
Improving Stream Data Quality With Protobuf Schema Validation
The requirements for fast and reliable data pipelines are growing quickly at Deliveroo as the business continues to grow and innovate.
Read more >
How to use Protobuf with Apache Kafka and Schema Registry
Full guide on working with Protobuf in Apache Kafka. Since Confluent Platform version 5.5, Avro is no longer the only schema in town....
Read more >
5 Reasons to Use Protocol Buffers Instead of JSON for Your ...
As you can see, by providing a schema, we now automatically get a class that can be used to encode and decode messages...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found