ccompatv6 not compatible with Confluent registry

We are currently migrating to Apicurio as our schema registry. During testing we had the following error

{
  "message": "No artifact with ID 'dz.is.stockexchange-ny-value' in group 'null' was found.",
  "error_code": 40401
}

This is only an issue if we register the schemas manually over the Apicurio UI by hand. If we use auto.register.schema it’s not an issue.

We tested this with 2.2.5.Final. We are using Kafkasql. We have the Legacy ID mode disabled.

Steps for reproduction

• register avro schema manually over UI group empty, artifact id = dz.is.stockexchange-ny-value and this avro definition

{
  "type": "record",
  "name": "FinancialSecurityValue",
  "namespace": "ch.xxx.integration",
  "doc": "This is a financial security record value in a fictitious stock exchange.",
  "fields": [
    {
      "name": "isin",
      "type": "string",
      "doc": "unique id of the financialsecurity"
    },
    {
      "name": "name",
      "type": "string",
      "doc": "name of the financial security"
    },
    {
      "name": "financialSecurityType",
      "type": "string",
      "doc": "type of the financial security, e.g. a stock"
    },
    {
      "name": "quote",
      "type": [
        "null",
        {
          "type": "string",
          "java-class": "java.math.BigDecimal"
        }
      ],
      "doc": "traded quote of this financial security",
      "default": null
    },
    {
      "name": "leverage",
      "type": [
        "null",
        {
          "type": "string",
          "java-class": "java.math.BigDecimal"
        }
      ],
      "doc": "leverage of an option, null for bonds and stocks",
      "default": null
    },
    {
      "name": "underlyingInstrumentIsin",
      "type": [
        "null",
        "string"
      ],
      "doc": "an option bases on an underlying financial instrument, e.g. a stock",
      "default": null
    },
    {
      "name": "optionType",
      "type": [
        "null",
        "string"
      ],
      "doc": "traded quote of this financial security",
      "default": null
    },
    {
      "name": "quoteEstimated",
      "type": "boolean",
      "doc": "if no trade occured marketmakers estimate quotes"
    },
    {
      "name": "quoteTime",
      "type": {
        "type": "long",
        "logicalType": "timestamp-millis"
      },
      "doc": "Timestamp (milliseconds since epoch) at which this quote got settled"
    },
    {
      "name": "averageVolume",
      "type": "int",
      "doc": "average number of stocks being traded on one day"
    },
    {
      "name": "exDividendDate",
      "type": [
        "null",
        {
          "type": "int",
          "logicalType": "date"
        }
      ],
      "doc": "date when dividend will be paid",
      "default": null
    },
    {
      "name": "interestRate",
      "type": [
        "null",
        {
          "type": "string",
          "java-class": "java.math.BigDecimal"
        }
      ],
      "doc": "interestRate of the bond in percent",
      "default": null
    }
  ]
}

• Use a spring app or the following call (issued by the spring app) to get the error

POST https://registry.com/subjects/dz.is.stockexchange-ny-value
Content-Type: application/vnd.schemaregistry.v1+json
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

{
  "schema": "{\"type\":\"record\",\"name\":\"FinancialSecurityValue\",\"namespace\":\"ch.xxx.integration\",\"doc\":\"This is a financial security record value in a fictitious stock exchange.\",\"fields\":[{\"name\":\"isin\",\"type\":\"string\",\"doc\":\"unique id of the financialsecurity\"},{\"name\":\"name\",\"type\":\"string\",\"doc\":\"name of the financial security\"},{\"name\":\"financialSecurityType\",\"type\":\"string\",\"doc\":\"type of the financial security, e.g. a stock\"},{\"name\":\"quote\",\"type\":[\"null\",{\"type\":\"string\",\"java-class\":\"java.math.BigDecimal\"}],\"doc\":\"traded quote of this financial security\",\"default\":null},{\"name\":\"leverage\",\"type\":[\"null\",{\"type\":\"string\",\"java-class\":\"java.math.BigDecimal\"}],\"doc\":\"leverage of an option, null for bonds and stocks\",\"default\":null},{\"name\":\"underlyingInstrumentIsin\",\"type\":[\"null\",\"string\"],\"doc\":\"an option bases on an underlying financial instrument, e.g. a stock\",\"default\":null},{\"name\":\"optionType\",\"type\":[\"null\",\"string\"],\"doc\":\"traded quote of this financial security\",\"default\":null},{\"name\":\"quoteEstimated\",\"type\":\"boolean\",\"doc\":\"if no trade occured marketmakers estimate quotes\"},{\"name\":\"quoteTime\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},\"doc\":\"Timestamp (milliseconds since epoch) at which this quote got settled\"},{\"name\":\"averageVolume\",\"type\":\"int\",\"doc\":\"average number of stocks being traded on one day\"},{\"name\":\"exDividendDate\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}],\"doc\":\"date when dividend will be paid\",\"default\":null},{\"name\":\"interestRate\",\"type\":[\"null\",{\"type\":\"string\",\"java-class\":\"java.math.BigDecimal\"}],\"doc\":\"interestRate of the bond in percent\",\"default\":null}]}"
}

[carlesarnal]

I think a global setting similar to the content ID legacy mode (as @dweber019 indicated) would be appropriate.

Note that we have another feature designed to address this. When we receive content we calculate two hashes: 1) SHA hash of the as-is content without modification and 2) SHA hash of the canonicalized version of the content

We actually go further than simply minifying the content. We attempt to actually canonicalize it. This might entail things like changing (sorting) the ordering of JSON properties where the order doesn’t matter. Unfortunately, I don’t think there is a way to utilize this feature when using the ccompat API.

This last part may change in the near future. In V7, confluent added the option to normalize schemas in their api, so we will be able to use our tooling to normalize the schemas. That said, doing so is an option in the API so it will be a tooling decision to use that parameter or not.

But this is another option we might consider: have a global option to use canonicalHash in the ccompat API rather than hash.

[carlesarnal]

My point is that the schema you’re manually uploading in the UI is actually “different” than the one being tested by the spring application.

This is the one uploaded manually in the UI:

"{\n  \"type\": \"record\",\n  \"name\": \"FinancialSecurityValue\",\n  \"namespace\": \"ch.xxx.integration\",\n  \"doc\": \"This is a financial security record value in a fictitious stock exchange.\",\n  \"fields\": [\n    {\n      \"name\": \"isin\",\n      \"type\": \"string\",\n      \"doc\": \"unique id of the financialsecurity\"\n    },\n    {\n      \"name\": \"name\",\n      \"type\": \"string\",\n      \"doc\": \"name of the financial security\"\n    },\n    {\n      \"name\": \"financialSecurityType\",\n      \"type\": \"string\",\n      \"doc\": \"type of the financial security, e.g. a stock\"\n    },\n    {\n      \"name\": \"quote\",\n      \"type\": [\n        \"null\",\n        {\n          \"type\": \"string\",\n          \"java-class\": \"java.math.BigDecimal\"\n        }\n      ],\n      \"doc\": \"traded quote of this financial security\",\n      \"default\": null\n    },\n    {\n      \"name\": \"leverage\",\n      \"type\": [\n        \"null\",\n        {\n          \"type\": \"string\",\n          \"java-class\": \"java.math.BigDecimal\"\n        }\n      ],\n      \"doc\": \"leverage of an option, null for bonds and stocks\",\n      \"default\": null\n    },\n    {\n      \"name\": \"underlyingInstrumentIsin\",\n      \"type\": [\n        \"null\",\n        \"string\"\n      ],\n      \"doc\": \"an option bases on an underlying financial instrument, e.g. a stock\",\n      \"default\": null\n    },\n    {\n      \"name\": \"optionType\",\n      \"type\": [\n        \"null\",\n        \"string\"\n      ],\n      \"doc\": \"traded quote of this financial security\",\n      \"default\": null\n    },\n    {\n      \"name\": \"quoteEstimated\",\n      \"type\": \"boolean\",\n      \"doc\": \"if no trade occured marketmakers estimate quotes\"\n    },\n    {\n      \"name\": \"quoteTime\",\n      \"type\": {\n        \"type\": \"long\",\n        \"logicalType\": \"timestamp-millis\"\n      },\n      \"doc\": \"Timestamp (milliseconds since epoch) at which this quote got settled\"\n    },\n    {\n      \"name\": \"averageVolume\",\n      \"type\": \"int\",\n      \"doc\": \"average number of stocks being traded on one day\"\n    },\n    {\n      \"name\": \"exDividendDate\",\n      \"type\": [\n        \"null\",\n        {\n          \"type\": \"int\",\n          \"logicalType\": \"date\"\n        }\n      ],\n      \"doc\": \"date when dividend will be paid\",\n      \"default\": null\n    },\n    {\n      \"name\": \"interestRate\",\n      \"type\": [\n        \"null\",\n        {\n          \"type\": \"string\",\n          \"java-class\": \"java.math.BigDecimal\"\n        }\n      ],\n      \"doc\": \"interestRate of the bond in percent\",\n      \"default\": null\n    }\n  ]\n}\n"

And this is the one being tested by the spring application, from your own POST request:

"{\"type\":\"record\",\"name\":\"FinancialSecurityValue\",\"namespace\":\"ch.xxx.integration\",\"doc\":\"This is a financial security record value in a fictitious stock exchange.\",\"fields\":[{\"name\":\"isin\",\"type\":\"string\",\"doc\":\"unique id of the financialsecurity\"},{\"name\":\"name\",\"type\":\"string\",\"doc\":\"name of the financial security\"},{\"name\":\"financialSecurityType\",\"type\":\"string\",\"doc\":\"type of the financial security, e.g. a stock\"},{\"name\":\"quote\",\"type\":[\"null\",{\"type\":\"string\",\"java-class\":\"java.math.BigDecimal\"}],\"doc\":\"traded quote of this financial security\",\"default\":null},{\"name\":\"leverage\",\"type\":[\"null\",{\"type\":\"string\",\"java-class\":\"java.math.BigDecimal\"}],\"doc\":\"leverage of an option, null for bonds and stocks\",\"default\":null},{\"name\":\"underlyingInstrumentIsin\",\"type\":[\"null\",\"string\"],\"doc\":\"an option bases on an underlying financial instrument, e.g. a stock\",\"default\":null},{\"name\":\"optionType\",\"type\":[\"null\",\"string\"],\"doc\":\"traded quote of this financial security\",\"default\":null},{\"name\":\"quoteEstimated\",\"type\":\"boolean\",\"doc\":\"if no trade occured marketmakers estimate quotes\"},{\"name\":\"quoteTime\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},\"doc\":\"Timestamp (milliseconds since epoch) at which this quote got settled\"},{\"name\":\"averageVolume\",\"type\":\"int\",\"doc\":\"average number of stocks being traded on one day\"},{\"name\":\"exDividendDate\",\"type\":[\"null\",{\"type\":\"int\",\"logicalType\":\"date\"}],\"doc\":\"date when dividend will be paid\",\"default\":null},{\"name\":\"interestRate\",\"type\":[\"null\",{\"type\":\"string\",\"java-class\":\"java.math.BigDecimal\"}],\"doc\":\"interestRate of the bond in percent\",\"default\":null}]}"

So even though they look similar and they might be equivalent, from a content POV, they’re actually different. If you get the first content, the one you just uploaded in the UI, and try to execute the request that the spring application is doing with that content, it will succeed (I just tested it locally).

Hope this makes sense.