[BUG]: Error while trying to enumerate a DataFrame containing a column of type date: The method or operation is not implemented.

Describe the bug An error occurred while trying to enumerate a DataFrame containing a column of type date.

To Reproduce Steps to reproduce the behavior:

public void Test()
{
  var dataFrame1 = spark.Session.Sql("SELECT * FROM VALUES ('2020-1-1'), ('2020-1-2') AS (DateAsString)");
  dataFrame1.Show();
  dataFrame1.PrintSchema();
  var rows1 = dataFrame1.Collect().Count(); // returns 2

  var dataFrame2 = spark.Session.Sql("SELECT * FROM VALUES CAST('2020-1-1' AS date), CAST('2020-1-2' AS date) AS (DateAsDate)");
  dataFrame2.Show();
  dataFrame2.PrintSchema();
  var rows2 = dataFrame2.Collect().Count(); // failed with "System.NotImplementedException" {"The method or operation is not implemented."}
}

Command-Line output: ±-----------+ |DateAsString| ±-----------+ | 2020-1-1| | 2020-1-2| ±-----------+

root |-- DateAsString: string (nullable = false)

±---------+ |DateAsDate| ±---------+ |2020-01-01| |2020-01-02| ±---------+

root |-- DateAsDate: date (nullable = true)

Exception “System.NotImplementedException: The method or operation is not implemented. at Microsoft.Spark.Sql.Row.Convert() at Microsoft.Spark.Sql.Row…ctor(Object[] values, StructType schema) at Microsoft.Spark.Sql.RowConstructor.GetRow() at Microsoft.Spark.Sql.RowCollector.Collect(ISocketWrapper socket)+MoveNext() at Microsoft.Spark.Sql.DataFrame.GetRows(String funcName)+MoveNext() at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source) at MyTest.Test() in …”

Expected behavior A DataFrame containing columns of type date can be enumerated.

Additional context Tested with a latest version (0.7.0) of the .net spark library.

[ravshMSFT]

Upgraded to 0.9.0 and this issue is resolved.

[elvaliuliuliu]

Can you try with Spark.NET v0.9.0? I think this should work with the new version.