Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
org.apache.spark.sql.streaming.StreamingQueryException: Maximum silent time must be at least 30000 milliseconds (AAD auth)
See original GitHub issueI’m using Spark 3.0.1, Scala 2.12 (Databricks Runtime 7.3 LTS). Basically the problem is as follows: I try to run Spark Structured Streaming job that reads from event hub using AAD auth. I got this error:
org.apache.spark.sql.streaming.StreamingQueryException: Maximum silent time must be at least 30000 milliseconds
I think the problem is here: https://github.com/Azure/azure-event-hubs-spark/blob/master/core/src/main/scala/org/apache/spark/eventhubs/client/ClientConnectionPool.scala#L65
in line 65:
val ehClientOption: EventHubClientOptions = new EventHubClientOptions().setMaximumSilentTime(ehConf.maxSilentTime.getOrElse(DefaultMaxSilentTime))
DefaultMaxSilentTime is set to SILENT_OFF: https://github.com/Azure/azure-event-hubs-spark/blob/master/core/src/main/scala/org/apache/spark/eventhubs/package.scala#L40
However it should be set to MinSilentTime as per documentation: https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.eventhubs.eventhubclientoptions.setmaximumsilenttime?view=azure-java-stable (Time must be at least SILENT_MINIMUM.)
So the correct line should like this:
val ehClientOption: EventHubClientOptions = new EventHubClientOptions().setMaximumSilentTime(ehConf.maxSilentTime.getOrElse(MinSilentTime))
Issue Analytics
- State:
- Created 3 years ago
- Comments:6 (3 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
That’s a bug in the validation of the underlying Java Event Hubs client. SILENT_OFF is absolutely supposed to be valid, but the test at https://github.com/Azure/azure-sdk-for-java/blob/6b9c7969be08cf42320c22198f9ebc3a0fbd2f35/sdk/eventhubs/microsoft-azure-eventhubs/src/main/java/com/microsoft/azure/eventhubs/EventHubClientOptions.java#L110 is broken.
We already have some other fixes in progress, so we can roll this into the next release. In the meantime, you should be able to work around this by configuring the silent time to be some large value.
@tfayyaz were you able to figure out the solution for your scala error? I am also facing the same issue. I am using latest version of Eventhub spark library: com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22 Even when I explicitly set 40000 ms as duration, it throws same error: java.lang.IllegalArgumentException: Maximum silent time must be at least 30000 milliseconds @JamesBirdsall /@nyaghma