Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Kafkajs - confluent schema registry error while consuming message from java avro producer (Magic byte)

See original GitHub issue

Hi I have a spring cloud producer which uses a confluent schema registry that generates and publishes to Kafka topic every 15 seconds. I can successfully consume messages from java consumers without any issue. I wrote a node js consumer using Kafka js, it shows error while consuming the message from topic

I have source code here. Please have a look and provide points to figure out the issue.

   "message":"[Consumer] Crash: KafkaJSNumberOfRetriesExceeded: Message encoded with magic byte {\"type\":\"Buffer\",\"data\":[72]}, expected {\"type\":\"Buffer\",\"data\":[0]}",
   "stack":"KafkaJSNonRetriableError\n  Caused by: ConfluentSchemaRegistryArgumentError: Message encoded with magic byte {\"type\":\"Buffer\",\"data\":[72]}, expected {\"type\":\"Buffer\",\"data\":[0]}\n    at SchemaRegistry.decode (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/@kafkajs/confluent-schema-registry/dist/SchemaRegistry.js:133:19)\n    at Runner.eachMessage (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/src/consumer.js:25:43)\n    at Runner.processEachMessage (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/consumer/runner.js:151:20)\n    at onBatch (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/consumer/runner.js:287:20)\n    at /Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/consumer/runner.js:339:21\n    at invoke (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:38:5)\n    at push (/Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:51:7)\n    at /Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:60:53\n    at new Promise (<anonymous>)\n    at /Users/ereshgorantla/Documents/Dev/My Work/node/kafka-consumer/node_modules/kafkajs/src/utils/concurrency.js:60:20"

I was wondered, kafka js compares magic byte

async decode(buffer) {
        if (!Buffer.isBuffer(buffer)) {
            throw new errors_1.ConfluentSchemaRegistryArgumentError('Invalid buffer');
        const { magicByte, registryId, payload } = wireDecoder_1.default(buffer);
        if (, magicByte) !== 0) {
            throw new errors_1.ConfluentSchemaRegistryArgumentError(`Message encoded with magic byte ${JSON.stringify(magicByte)}, expected ${JSON.stringify(wireEncoder_1.MAGIC_BYTE)}`);
        const schema = await this.getSchema(registryId);
        return schema.fromBuffer(payload);

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:1
  • Comments:7

github_iconTop GitHub Comments

zwerg44commented, Nov 15, 2021

Maybe it will save someone couple of minutes. Faced the similar traceback about Magic byte today.

In my case:

  • message.value - was Avro-encoded
  • message.key - was just plaintext

So, you might want to check your keys (or values) are actually Avro-encoded : message.key.toString()

clodiocommented, Jan 27, 2022

same probleme to me. Due to this issue , i can not use kafkajs

Read more comments on GitHub >

github_iconTop Results From Across the Web

Formats, Serializers, and Deserializers
New Kafka serializers and deserializers are available for Protobuf and JSON Schema, along with Avro. The serializers can automatically register schemas when ......
Read more >
Unknown magic byte with kafka-avro-console-consumer
If your message has not been serialized using the Schema Registry serializer, ... it with it, and will get the Unknown magic byte!...
Read more >
Confluent 3.2.2 - Avro Consumer not able to ... - Google Groups
i've a java consumer program which is publishing Avro messages to Kafka Topic. ... The error is about a missing "magic byte" because...
Read more >
Usage with KafkaJS · Confluent schema registry
Although Confluent Schema Registry can be used with any Kafka client, ... of an application that consumes from a topic of AVRO encoded...
Read more >
Integrating with AWS Glue Schema Registry
Use case: Connecting Schema Registry to Amazon MSK or Apache Kafka. Let's assume you are writing data to an Apache Kafka topic, and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found