Error creating DStream on Azure Databricks

Hi,

Following this example I’m trying to create EventHub DStream but running into the exception below. Here is my code;

val ssc = new StreamingContext(sc, Seconds(batchIntervalSeconds))
ssc.checkpoint(checkpointDir)
  
val inputDStream = EventHubsUtils.createDirectStreams(
  ssc,
  eventhubsNamespace,
  progressDir,
  Map(eventhubsName -> eventhubsParameters))

• Actual behavior

java.lang.NoSuchMethodError: org.apache.spark.streaming.eventhubs.EventHubDirectDStream$.$lessinit$greater$default$5()Lscala/Function5;
	at org.apache.spark.streaming.eventhubs.EventHubsUtils$.createDirectStreams(EventHubsUtils.scala:52)

• Expected behavior DStream is successfully created

• Spark version Databricks Runtime 3.5 Apache Spark 2.2.1 Scala 2.11

• spark-eventhubs artifactId and version com.microsoft.azure:azure-eventhubs-spark_2.11:jar:2.1.6

• Dependencies automatically pulled by Databricks proton-j-0.22.0.jar slf4j-api-1.7.25.jar

[sabeegrewal]

Ah yes, you’re right. Glad it’s working! I’ll be updating documentation over the next couple weeks so this process is a little clearer.

[syedhassaanahmed]

Hi @sabeegrewal I’ve peeked into the eventhubs-databricks.jar and the correct namespace there is org.apache.spark.eventhubs.common.EventHubsUtils. Thanks for the help!