Can't get spark-streaming-eventhubs_2.11 to work with Jupyter in HDInsight 3.6 (Spark 2.1)

I can launch spark-shell with

spark-shell --packages com.microsoft.azure:spark-streaming-eventhubs_2.11:2.0.5

which works fine, but when I use the same package in Jupyter with

%%configure -f
{ "conf": {"spark.jars.packages": "com.microsoft.azure:spark-streaming-eventhubs_2.11:2.0.5" }}

It always fails with the cryptic error:

The code failed because of a fatal error: Session 31 unexpectedly reached final status ‘dead’. See logs:

Or sometimes with this one:

The code failed because of a fatal error: Status ‘shutting_down’ not supported by session…

Also interesting that streaming-eventhubs_2.10 does work in Jupyter

[romitgirdhar]

Done! Just submitted a PR with a walkthrough of using the JAR in Jupyter using the PySpark3 kernel. Please review and approve at your earliest convenience.

[syedhassaanahmed]

@sabeegrewal This is how I imported the package in Jupyter notebook, HDInsight

%%configure -f
{
    "conf": { 
        "spark.jars.packages": "com.microsoft.azure:azure-eventhubs-spark_2.11:2.1.6",
        "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.11"
    }
}