Writing into EvenHubs, Spark executors fail silently when getting throttled

I have a job running in databricks which writes the result of a batch query (which returns 7 millions rows) into an eventhub with 1 TU. This job completes successfully but when I inspect the eventhub I see only around 2million events getting published.

On further inspection of the executor logs I see com.microsoft.azure.eventhubs.ServerBusyException being thrown but this error does not abort the spark job. Therefore its possible that the write operation returns without any exception but still fail to publish all the events.

Sample code:

val ehWriteConf = EventHubsConf(peersEhConnStr)
val peersLAFrame = peersFrame.select($"body", $"properties")
peersLAFrame.write
               .format("eventhubs")
               .options(ehWriteConf.toMap)
               .save()

Bug Report:

• Actual behavior Throttled eventhub write/ingress operations fail silently.
• Expected behavior When the eventhub operation is throttled that error should possibly abort the job.
• Spark version Databricks 5.5 LTS (includes Apache Spark 2.4.3, Scala 2.11)
• spark-eventhubs artifactId and version azure-eventhubs-spark_2.11:2.3.13

[rvoak]

@nyaghma Yes. Just tested it; throws an exception with 2.3.16!

[nyaghma]

@rvoak Thanks for letting us know. Can you please try again with the version 2.3.16? You should be able to see the exception in this version. Please let me know if the issue still happens with version 2.3.16.