Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Deserialization exception: Unknown magic byte!
See original GitHub issueHi, I’m using version 1.1.0 of this connector and I’m trying to export some data from a Kafka topic to Google Big Query. I verified that I have a correct schema in my schema registry and that I can consume using the kafka-avro-console-consumer (from Confluent platform).
When starting the connector, however, I’m getting the following exceptions in the log:
Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1 Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
The table is created in BigQuery and the fields look correct, but no rows are being written as a result of the exception.
Any pointers?
Issue Analytics
- State:
- Created 5 years ago
- Comments:7 (3 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Hi, I am getting the same error. I converted a JSON file into an AVRO file.
@DediBar KCBQ currently works out of the box with Avro topics, not CSV/JSON topics. The reason for this is that we have to map from incoming message schemas to BigQuery schemas (e.g. from Avro schema to BigQuery schema). We have made this mapper pluggable, but we only support Avro out of the box. If you want to use JSON/CSV input rather than Avro input, you’ll have to write your own.
https://github.com/wepay/kafka-connect-bigquery/blob/master/kcbq-api/src/main/java/com/wepay/kafka/connect/bigquery/api/SchemaRetriever.java