Improve compatibility with the Confluent Platform

Currently, Spring Cloud Stream provides a dedicated client for Confluent’s Schema Registry. But because the Avro serializer used by Confluent’s Kafka REST Proxy encodes the Schema-Id into the message payload, messages sent by the REST Proxy are not compatible with SCS’s AvroSchemaRegistryClientMessageConverter.

As decribed in their mailing list, Confluent’s serializer perfaces the actual Avro message with the Id of the message’s Schema in their Schema Registry. Therefore, the MessageConverter would need to extract the Schema-Id before deserializing the actual payload in a way similar to their AbstractKafkaAvroDeserializer:

      ByteBuffer buffer = getByteBuffer(payload);
      id = buffer.getInt();
      String subject = includeSchemaAndVersion ? getSubjectName(topic, isKey) : null;
      Schema schema = schemaRegistry.getBySubjectAndID(subject, id);
      int length = buffer.limit() - 1 - idSize;

While trying to get SCS to work with the Confluent Platform, we already implemented a ConfluentSchemaRegistryClientMessageConverter, but an official implementation would be preferable of course.

[viniciusccarvalho]

Thank you for your interest, we have this as a feature in the backlog, but just not a priority ATM.

One area of concern is that we also modify the payload to embed headers, and the first byte is reserved for the version of the headerMode. I’m not sure how would allowing reading from confluent’s format could interfere there, unless you always use raw mode for headers.

Right now we are revisiting the entire serialization/contentType story inside the framework, we will be on a better position to support this once 2.0 gets to a stable release. Then we may be able to create a toggle feature for this.

Regards

[olegz]

Giving that this issue is a bit dated and the fact that schema registry moved to it’s own repo, I am closing it here. However it doesn’t mean the issue os not valid, but perhaps it needs to be refreshed and resubmitted