Same schema registered to multiple subjects - multiple global IDs - different behavior compared to Confluent Schema Registry

Consider the following example schema:

{"schema": "{\"type\": \"record\", \"name\": \"ID\", \"namespace\": \"com.example\", \"fields\": [{\"doc\": \"ID\", \"name\": \"id\", \"type\": \"int\"}]}"}

If I register this with Confluent Schema Registry, on subject “test”, I get the following result:

$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8081/subjects/test/versions' -H 'Content-Type: application/vnd.schemaregistry.v1+json' [e-mail] ; echo
{"id":1}

Now, if I re-register it but on a different subject “test2”:

$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8081/subjects/test2/versions' -H 'Content-Type: application/vnd.schemaregistry.v1+json' [e-mail] ; echo
{"id":1}

We can see that the schema is now registered on two different subjects:

$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8081/subjects/test/versions' ; echo
[1]
$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8081/subjects/test/versions/1' ; echo
{"subject":"test","version":1,"id":1,"schema":"{\"type\":\"record\",\"name\":\"ID\",\"namespace\":\"com.example\",\"fields\":[{\"name\":\"id\",\"type\":\"int\",\"doc\":\"ID\"}]}"}
$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8081/subjects/test2/versions' ; echo
[1]
$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8081/subjects/test2/versions/1' ; echo
{"subject":"test2","version":1,"id":1,"schema":"{\"type\":\"record\",\"name\":\"ID\",\"namespace\":\"com.example\",\"fields\":[{\"name\":\"id\",\"type\":\"int\",\"doc\":\"ID\"}]}"}

As we can see - even though the schema is registered on two different subjects, it has the same Global ID: 1

However, Apicurio Registry 1.3.0.Final, and I’ve tested this with both the asyncmem and the streams backend, behaves differently:

$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8080/api/ccompat/subjects/test/versions' -H 'Content-Type: application/vnd.schemaregistry.v1+json' [e-mail] ; echo
{"id":1}
$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8080/api/ccompat/subjects/test2/versions' -H 'Content-Type: application/vnd.schemaregistry.v1+json' [e-mail] ; echo
{"id":2}

We can see already that the global ID is not the same, and this can be verified by using the versions endpoint:

$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8080/api/ccompat/subjects/test/versions/1' ; echo
{"id":1,"subject":"test","version":1,"schema":"{\"type\": \"record\", \"name\": \"ID\", \"namespace\": \"com.example\", \"fields\": [{\"doc\": \"ID\", \"name\": \"id\", \"type\": \"int\"}]}"}
$ curl  -H 'Accept: application/vnd.schemaregistry.v1+json' 'http://localhost:8080/api/ccompat/subjects/test2/versions/1' ; echo
{"id":2,"subject":"test2","version":1,"schema":"{\"type\": \"record\", \"name\": \"ID\", \"namespace\": \"com.example\", \"fields\": [{\"doc\": \"ID\", \"name\": \"id\", \"type\": \"int\"}]}"}

Is this a bug, or by design? If it’s by design, why?

For me, having an ID that is the same for a specific schema, regardless of subject is part of the beauty of using a Schema Registry, as it gives me flexibility in how I use schemas and route data.

[EricWittmann]

Update: this will be addressed in version 2.0.

@carlesarnal and @famartinrh - can you guys collaborate on making sure this is fixed? I think there are two things to do here:

1. The ccompat API implementation needs to be updated to use contentId instead of globalId (I believe in all cases) ( @carlesarnal )
2. The Apicurio Registry Serdes classes needs a configuration option to read/write the contentId instead of the globalId ( @famartinrh )

[EricWittmann]

We had a quick internal discussion about this today and realized that we don’t necessarily have to implement this functionality for the Apicurio Registry API - we just need to implement it for the Confluent Compatibility API. Thanks for that suggestion @jsenko

I need to think through how we might be able to provide a separate “globalId” for the Confluent compatibility layer that behaves the way you are expecting and what implications that might have for other parts of the application and other use-cases. For example, does this approach break the ability to mix and match Apicurio serdes classes with Confluent serdes classes?

So this is just an idea at this point, but if we can make it work then we’ll have an option that will allow us to support the feature without doing a 2.0 release.