Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add message header to EventData writing to EventHub

See original GitHub issue

This is a feature request to allow sending message headers along with the message body when streaming to an EventHub sink. In our system the backend relies on the message headers, so we needed to implement our own ForeachWriter in Spark, just to add the message header to the EventData object. In our custom ForeachWriter we do it like this

import{EventData, EventHubClient}
val data: EventData = EventData.create(eventByteArray)
data.getProperties.put("EventType", messageHeader)

Having our own ForeachWriter we can not benefit from all the optimizations you put into this library. We would be very happy to be able to use a syntax like this:

// Write body data from a DataFrame to EventHubs with a message header
val ds = df
  .selectExpr("header", "body")
  .options(ehWriteConf.toMap)    // EventHubsConf containing the destination EventHub connection string.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

sabeegrewalcommented, Aug 3, 2018

@jpsimen yea, adding a single String, String tuple doesn’t make much sense to me. I’m going to add it as a Map[String, String] - you should be able to have multiple properties per EventData 👍

jpsimencommented, Aug 3, 2018

@sabeegrewal no objections to adding it to the EventHubsSink. 😃 Regarding your question: Yes, a key value pair of type (String, String) to set the EventData properties would be nice.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Sending header data in Azure eventhub using Python
In this I am trying to send properties in the form of a dict along with message body , the below is the...
Read more >
Send event - Microsoft Learn
Request Headers. See Common parameters and headers for headers and parameters that are used by all requests related to Event Hubs.
Read more >
5 Message handling with Event Hubs - liveBook · Manning
An Event Hub ingests messages from applications. It records the details of each message in a journal and saves the message data for...
Read more >
Azure Event Hubs - Apache Camel
Send and receive events to/from Azure Event Hubs using AMQP protocol. ... Async Consumer and Producer; Usage; Message Headers; Message body type ...
Read more >
Azure.Messaging.EventHubs 5.7.5 - NuGet
Install the package. Install the Azure Event Hubs client library for .NET with NuGet: dotnet add package Azure.Messaging.EventHubs ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found