Empty string is converted to null

spark-redshift (v3.0.0-preview1) will convert an empty string '' into a null value when reading data from redshift:

spark.read
  .format("com.databricks.spark.redshift")
  .option("url", url)
  .option("aws_iam_role", iamRole)
  .option("tempdir", tmpDir)        
  .option("query", "select '' as foo, null as bar")
  .load()
  .show()

[info] +----+----+
[info] | foo| bar|
[info] +----+----+
[info] |null|null|
[info] +----+----+

In redshift, there is a distinction between null and '':

redshift-user=> select foo is null as foonull, bar is null as barnull from (select '' as foo, null as bar);
 foonull | barnull 
---------+---------
 f       | t
(1 row)

spark-sql also supports this distinction:

spark.sql("select '' as foo, null as bar").show()

[info] +---+----+
[info] |foo| bar|
[info] +---+----+
[info] |   |null|
[info] +---+----+

[drewrobb]

Databricks has decided to close source new work on this project, so I don’t expect any update or any PR accepted here. Maybe a community fork will get going, I’m disappointed and will avoid Databricks open source projects going forward.

[Enigma-v]

I was wondering if you had an issue with Redshift that returns NULL in one database and on the same dataset but in different database it returns empty string. The data is the same on both of the databases?