Kafkajs - confluent schema registry error while consuming message from java avro producer (Magic byte)

Hi I have a spring cloud producer which uses a confluent schema registry that generates and publishes to Kafka topic every 15 seconds. I can successfully consume messages from java consumers without any issue. I wrote a node js consumer using Kafka js, it shows error while consuming the message from topic

I have source code here. Please have a look and provide points to figure out the issue.

{
   "level":"ERROR",
   "timestamp":"2021-08-30T09:36:01.664Z",
   "logger":"kafkajs",
   "message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Message encoded with magic byte {\"type\":\"Buffer\",\"data\":[72]}, expected {\"type\":\"Buffer\",\"data\":[0]}",
   "groupId":"test-group",
   "retryCount":5,
   "stack":"KafkaJSNonRetriableError\n  Caused by: ConfluentSchemaRegistryArgumentError: Message encoded with magic byte {\"type\":\"Buffer\",\"data\":[72]}, expected {\"type\":\"Buffer\",\"data\":[0]}\n    at SchemaRegistry.decode (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/@kafkajs/confluent-schema-registry/dist/SchemaRegistry.js:133:19)\n    at Runner.eachMessage (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/src/consumer.js:25:43)\n    at Runner.processEachMessage (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/consumer/runner.js:151:20)\n    at onBatch (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/consumer/runner.js:287:20)\n    at /Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/consumer/runner.js:339:21\n    at invoke (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:38:5)\n    at push (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:51:7)\n    at /Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:60:53\n    at new Promise (<anonymous>)\n    at /Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:60:20"
}

I was wondered, kafka js compares magic byte

async decode(buffer) {
        if (!Buffer.isBuffer(buffer)) {
            throw new errors_1.ConfluentSchemaRegistryArgumentError('Invalid buffer');
        }
        const { magicByte, registryId, payload } = wireDecoder_1.default(buffer);
        if (Buffer.compare(wireEncoder_1.MAGIC_BYTE, magicByte) !== 0) {
            throw new errors_1.ConfluentSchemaRegistryArgumentError(`Message encoded with magic byte ${JSON.stringify(magicByte)}, expected ${JSON.stringify(wireEncoder_1.MAGIC_BYTE)}`);
        }
        const schema = await this.getSchema(registryId);
        return schema.fromBuffer(payload);
    }

[zwerg44]

Maybe it will save someone couple of minutes. Faced the similar traceback about Magic byte today.

In my case:

• message.value - was Avro-encoded
• message.key - was just plaintext

So, you might want to check your keys (or values) are actually Avro-encoded : message.key.toString()

[clodio]

same probleme to me. Due to this issue , i can not use kafkajs