question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Kafka] Allow Json consume to natively read AVRO Message

See original GitHub issue

Context

For my organization project, i needed to produce and consume AVRO messages from the Schema Registry so i develop a custom KakfaClient to produce the AVRO message.

Concerning the consumption, i detected after a debugging phase that the actual KafkaReceiver is almost able to treat an incoming AVRO message format thanks to the readJson method

Here is the actual implementation

            ConsumerJsonRecord jsonRecord = new ConsumerJsonRecord(thisRecord.key(), null, valueNode, headersMap);

Here is the code which allow parsing AVRO record

            jsonRecords.add(new ConsumerJsonRecord(keyNode, null, valueNode, headersMap));

with keyNode

            JsonNode keyNode = objectMapper.readTree(key.toString());

I can do a PR if this issue is accepted

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:2
  • Comments:6 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
nirmalchandracommented, Apr 28, 2021

Thanks @M3lkior. This looks great. Yes please, please go ahead with the PR and assign the following folks as reviewers please. @sparrowV , @respondsid , @dandalavinod, @ddingel, @santhoshTpixler

Will assign additional reviewers if needed.

1reaction
authorjappscommented, May 31, 2021

Released. Available in v 1.3.28

Read more comments on GitHub >

github_iconTop Results From Across the Web

Consume single json but publish multiple AVRO messages
I recently setup a project and tried kafka stream APIs to publish AVRO. Requirement: Need to parse a complex Jsonobject which might have...
Read more >
Optimizing Data Storage Using Compact Binary Formats
Once your data is in Avro format, Kafka Connect provides support to read this Avro data and write it in either Avro or...
Read more >
Avro Schema Serializer and Deserializer
Start Confluent Platform using the following command: · Verify registered schema types. · Use the producer to send Avro records in JSON as...
Read more >
Reading Avro Streams from Confluent Cloud into Apache Druid
Apache Druid can natively consume data out of a Kafka topic. ... In order to parse Avro messages, you first have to enable...
Read more >
Data formats - Apache Druid
Apache Druid can ingest denormalized data in JSON, CSV, or a delimited form such ... use a message header that allows the parser...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found