Add message header to EventData writing to EventHub

This is a feature request to allow sending message headers along with the message body when streaming to an EventHub sink. In our system the backend relies on the message headers, so we needed to implement our own ForeachWriter in Spark, just to add the message header to the EventData object. In our custom ForeachWriter we do it like this

import com.microsoft.azure.eventhubs.{EventData, EventHubClient}
...
val data: EventData = EventData.create(eventByteArray)
data.getProperties.put("EventType", messageHeader)

Having our own ForeachWriter we can not benefit from all the optimizations you put into this library. We would be very happy to be able to use a syntax like this:

// Write body data from a DataFrame to EventHubs with a message header
val ds = df
  .selectExpr("header", "body")
  .writeStream
  .format("eventhubs")
  .options(ehWriteConf.toMap)    // EventHubsConf containing the destination EventHub connection string.
  .start()

[sabeegrewal]

@jpsimen yea, adding a single String, String tuple doesn’t make much sense to me. I’m going to add it as a Map[String, String] - you should be able to have multiple properties per EventData 👍

[jpsimen]

@sabeegrewal no objections to adding it to the EventHubsSink. 😃 Regarding your question: Yes, a key value pair of type (String, String) to set the EventData properties would be nice.