Incorrect filtering by enqueuedTime

Bug Report:

Event Hub’s filtering by startingPosition and endingPosition seem to not work correctly and return events with EnqueuedTime that are out of bounds.

Actual behavior

By running the following code sample (credentials redacted)

val spark = SparkSession.builder().appName("AbrisPoc").master("local[2]").config("spark.sql.session.timeZone", "UTC").getOrCreate()

  val df =
    spark.read.format("eventhubs").option(
      "eventhubs.connectionString", "Endpoint=amqps://XXXXXXXXXX.servicebus.windows.net/XXXXXXXXXX;EntityPath=XXXXXXXXXXXXXXXXXXX;etc."
    ).option(
      "eventhubs.startingPosition", """{"offset": null, "seqNo": -1, "enqueuedTime": "2019-04-26T00:00:00.000000Z", "isInclusive": true}"""
    ).option(
      "eventhubs.endingPosition", """{"offset": null, "seqNo": -1, "enqueuedTime": "2019-04-26T23:59:59.999999Z", "isInclusive": true}"""
    ).load()
  df.selectExpr("""MAX(EnqueuedTime)""").show(10, truncate = false)

We get the following result

  +-----------------------+
  |max(EnqueuedTime)      |
  +-----------------------+
  |2019-04-27 00:13:00.889|
  +-----------------------+

which is outside the specified interval of 2019-04-26T00:00:00.000000Z to 2019-04-26T23:59:59.999999Z

Expected behavior

All the returned events should have an EnqueuedTime between the specified bounds.

Spark version :

2.3.3

spark-eventhubs artifactId and version:

from my sbt project:

libraryDependencies += "com.microsoft.azure" % "azure-eventhubs-spark_2.11" % "2.3.9"
libraryDependencies += "com.microsoft.azure" % "azure-eventhubs" % "2.2.0"

[guoch]

endingPosition / endingPositions determine the ending position of a batch query. Sine in a streaming query there is no end position, this value would be simply ignored even if it’s been set in the client code. If you want to read events to a certain point use batch query instead of streaming query.

Hi @nyaghma Thank you a lot for this reply and confirmation. If possible, I would suggest you could add the comment here as well https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/structured-streaming-eventhubs-integration.md#eventhubsconf in the end position, so it may help others who had the similar question like me.

[sandeep8530]

@nyaghma , Thanks a lot for looking into it. Yes, It is working fine, If i use only 3 digits millisecond precision.

However in this documentation (https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/PySpark/structured-streaming-pyspark.md) below code is used in the filter which gives 6 digits. Appreciate, if you could update this to avoid any further confusion.

endTime = dt.now().strftime(“%Y-%m-%dT%H:%M:%S.%fZ”)