Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

org.apache.spark.sql.streaming.StreamingQueryException: Maximum silent time must be at least 30000 milliseconds (AAD auth)

See original GitHub issue

I’m using Spark 3.0.1, Scala 2.12 (Databricks Runtime 7.3 LTS). Basically the problem is as follows: I try to run Spark Structured Streaming job that reads from event hub using AAD auth. I got this error: org.apache.spark.sql.streaming.StreamingQueryException: Maximum silent time must be at least 30000 milliseconds I think the problem is here: in line 65:

val ehClientOption: EventHubClientOptions = new EventHubClientOptions().setMaximumSilentTime(ehConf.maxSilentTime.getOrElse(DefaultMaxSilentTime))

DefaultMaxSilentTime is set to SILENT_OFF:

However it should be set to MinSilentTime as per documentation: (Time must be at least SILENT_MINIMUM.)

So the correct line should like this:

val ehClientOption: EventHubClientOptions = new EventHubClientOptions().setMaximumSilentTime(ehConf.maxSilentTime.getOrElse(MinSilentTime))

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

JamesBirdsallcommented, Mar 10, 2021

That’s a bug in the validation of the underlying Java Event Hubs client. SILENT_OFF is absolutely supposed to be valid, but the test at is broken.

We already have some other fixes in progress, so we can roll this into the next release. In the meantime, you should be able to work around this by configuring the silent time to be some large value.

parthjinwalacommented, Nov 24, 2022

@tfayyaz were you able to figure out the solution for your scala error? I am also facing the same issue. I am using latest version of Eventhub spark library: Even when I explicitly set 40000 ms as duration, it throws same error: java.lang.IllegalArgumentException: Maximum silent time must be at least 30000 milliseconds @JamesBirdsall /@nyaghma

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot connectivity issues - Azure Event Hubs
This article provides information on troubleshooting connectivity issues with Azure Event Hubs.
Read more >
Writing large DataFrame from PySpark to Kafka runs into timeout
option("", 30000) and different values but that didn't seem to have any effect. I'm running this in an Azure Databricks ...
Read more >
Azure Event Hubs | Cribl Docs
Value must be lower than rebalanceTimeout . Defaults to 30000 ms, i.e., 30 seconds. Rebalance timeout (ms): Maximum allowed time for each worker...
Read more >
Structured Streaming + Kafka Integration Guide (Kafka broker ...
groupId = org.apache.spark artifactId = spark-sql-kafka-0-10_2.12 version = 3.0.0-preview2. Please note that to use the headers functionality, your Kafka ...
Read more >
Microsoft Azure | InsightIDR Documentation - Docs @ Rapid7
Collect Azure Active Directory events to offer ingress authentication, single sign-on (SSO), cloud service activity, and cloud service admin activity detections ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found