Spark can't read Iceberg table created from Presto

Spark can’t read table which was created in Presto

create table iceberg.examples.test_table 
with (format = 'parquet')
as select timestamp '2021-01-19 23:59:59.999999' as ts;

In Spark

spark.read
      .format("iceberg")
      .load("examples.test_table")
      .show(10, false)

Fails with exception java.lang.UnsupportedOperationException: Spark does not support timestamp without time zone fields

Please add support of Timestamp without timezone type into Iceberg Spark runtime, because Iceberg supports this type

[davseitsev]

According to Spark docs TimestampType represents values comprising values of fields year, month, day, hour, minute, and second, with the session local time-zone. The timestamp value represents an absolute point in time. So it’s more like LocalDateTime. Why can’t Iceberg expose timestamp without timezone as Spark timestamp?

As far as I understand the problem is that Spark has only one timestamp type but Iceberg has two types which causes some inconsistency in type mapping between Spark and Iceberg.

In my opinion it could be solved in a following way:

1. For read access there is no problem, both Iceberg types (timestamp, timestamptz) could be exposed as Spark timestamp.
2. For insert there is no problem because we know target type defined in the table metadata and can choose correct one
3. For table creation we have inconsistency which could be solved by configuration property. Spark has such kind of property to choose how to store timestamp fields in parquet (spark.sql.parquet.outputTimestampType=(INT96 | TIMESTAMP_MICROS | TIMESTAMP_MILLIS)). Why can’t we have such logic for Iceberg?

[davseitsev]

Related issues: https://github.com/apache/iceberg/issues/2388 and https://github.com/apache/iceberg/issues/2244