org.apache.spark.sql.streaming.StreamingQueryException: Maximum silent time must be at least 30000 milliseconds (AAD auth)
See original GitHub issueI’m using Spark 3.0.1, Scala 2.12 (Databricks Runtime 7.3 LTS). Basically the problem is as follows: I try to run Spark Structured Streaming job that reads from event hub using AAD auth. I got this error:
org.apache.spark.sql.streaming.StreamingQueryException: Maximum silent time must be at least 30000 milliseconds
I think the problem is here: https://github.com/Azure/azure-event-hubs-spark/blob/master/core/src/main/scala/org/apache/spark/eventhubs/client/ClientConnectionPool.scala#L65
in line 65:
val ehClientOption: EventHubClientOptions = new EventHubClientOptions().setMaximumSilentTime(ehConf.maxSilentTime.getOrElse(DefaultMaxSilentTime))
DefaultMaxSilentTime is set to SILENT_OFF: https://github.com/Azure/azure-event-hubs-spark/blob/master/core/src/main/scala/org/apache/spark/eventhubs/package.scala#L40
However it should be set to MinSilentTime as per documentation: https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs.eventhubclientoptions.setmaximumsilenttime?view=azure-java-stable (Time must be at least SILENT_MINIMUM.)
So the correct line should like this:
val ehClientOption: EventHubClientOptions = new EventHubClientOptions().setMaximumSilentTime(ehConf.maxSilentTime.getOrElse(MinSilentTime))
Issue Analytics
- State:
- Created 3 years ago
- Comments:6 (3 by maintainers)
That’s a bug in the validation of the underlying Java Event Hubs client. SILENT_OFF is absolutely supposed to be valid, but the test at https://github.com/Azure/azure-sdk-for-java/blob/6b9c7969be08cf42320c22198f9ebc3a0fbd2f35/sdk/eventhubs/microsoft-azure-eventhubs/src/main/java/com/microsoft/azure/eventhubs/EventHubClientOptions.java#L110 is broken.
We already have some other fixes in progress, so we can roll this into the next release. In the meantime, you should be able to work around this by configuring the silent time to be some large value.
@tfayyaz were you able to figure out the solution for your scala error? I am also facing the same issue. I am using latest version of Eventhub spark library: com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22 Even when I explicitly set 40000 ms as duration, it throws same error: java.lang.IllegalArgumentException: Maximum silent time must be at least 30000 milliseconds @JamesBirdsall /@nyaghma