pyspark - timestamp with microseconds, causes exception on .save()

When column contains timestamp with non 0 microseconds the .save() fails with generic “com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.” exception.

Truncation of microsecond to 0 works around the problem. The output table created by the .save() has column of “datetime” data type. Hence i presume it might be related to handling of “rounding” microseconds to satisfy precision requirements of “datetime” datatype.

env:

• sql spark connector version 1.1,
• spark 2.4.5
• databricks 6.4 runtime.

how to reproduce:

batchTimestamp = datetime.now()

#
# uncomment to truncate milliseconds, only truncation to 0 works
#batchTimestamp = batchTimestamp.replace(microsecond = 0)

print(batchTimestamp.isoformat(sep=' '))

df = spark \
  .createDataFrame([("a", 1), ("b", 2), ("c",  3)], ["Col1", "Col2"]) \
  .withColumn('ts', lit(batchTimestamp))

df.show()

df \
  .write \
  .format("com.microsoft.sqlserver.jdbc.spark") \
  .mode("overwrite") \
  .option("url", sql_url) \
  .option("dbtable", 'test_table') \
  .option("user", sql_username) \
  .option("password", sql_password) \
  .save()

[rajmera3]

As @shivsood mentioned this issue occurs due to the mapping of timestamp to DateTime instead of DateTime2. As mentioned in #83 the issue is with datetime2(0) but datetime2(x) works.

This is not specific to the connector itself and a PR will be made to Spark for a fix. We will update this issue once that is created

[luxu1-ms]

@chopraarjun I think it might be another issue there. If your date format only has 3 digits of microseconds, use both datetime and datetimme2 will not raise errors related to #39. Can you please open a new issue and add details about your environment and repro, thank you.