Preparing for Protobuf and JSON Schema support

The confluent schema registry as of version 5.5 supports 2 new schema types, Protobuf and Json Schema, see this article. In order to ultimately support these new schema types I would like to make some proposals to the users/contributors of this library.

Goals Separate specifics of the Avro schema/serdes from internals of the library. (Cache, Wire Encoder/Decoder, Subject generation) Maintain backwards compatibility with older Schema Registry deployments.

Separate Avro from internals

Cache - change the cache to just store the schema string, since other schema types may not be in Json Wire Encoder/Decoder - change the encoder/decoder to just do the wire encoding/decoding Subject Generation - delegate subject generation to specific schema serdes, and future iteration implement the different strategies

All of these changes should push the library to create a Serdes interface that might look something like this.

interface Serdes {
  serialize(schema: ConfluentSchema, payload: any) : Buffer
  deserialize(schema: ConfluentSchema, buffer: Buffer) : any 
}

Backwards Compatibility

In version 5.5.0 the schema registry introduces a new field to the various apis called schemaType. Reference here. Since the default is AVRO, and older versions fail when using this field, if the client uses AVRO schema/serdes it should omit the field.

There is a potential for breaking Api changes in this refactor, but users should not need to upgrade their Schema Registry deployment.

Note: I have a rough draft branch of some of these ideas, and want to get feedback before moving forward

[Malkiz]

Hi @dskatz22 and @Nevon , What is the status of this issue?

@dskatz22 I have read the code changes in your branch, and I think I understand it. I would be happy to continue the work you did to add support for Protobuf and JSON Schema.

I just wanted to make sure that this direction is considered acceptable by the library authors/maintainers ( @erikengervall ? ).

Thanks!

[Nevon]

While I’m not the primary author of this library, and am hesitant to take on responsibility for it (hello @erikengervall 👋), I’d be happy to take a look at a pull request and provide feedback.

Curse my previous self!

I’m currently swamped with being the primary maintainer of KafkaJS along with my day job, so I don’t feel like I have the bandwidth to also actively maintain @kafkajs/confluent-schema-registry and cannot commit to helping driving this change through.

In terms of general thoughts, with JSON Schema and Protobuf being officially supported in Confluent schema registry, it makes perfect sense that we should also do so. I can’t speak to @dskatz22’s implementation, as I don’t have the necessary background on how this is implemented on the registry side and how other clients are working with it.