Timestamp not parsed correctly on Athena

The data type of the timestamp columns are flawed inspite of being correct in the DataFrame. The schema of the DataFrame is:

root
 |-- branch_id: long (nullable = true)
 |-- comment: string (nullable = true)
 |-- created_at: timestamp (nullable = true)
...

The output of the DataFrame show():

--------------+--------------------+-----------------------+
|branch_id|             comment|         created_at|
--------------+--------------------+-----------------------+
|    13501|                    |2017-05-09 08:21:35|
|    14081|                    |2017-05-09 08:53:29|
...
--------------+--------------------+-----------------------+

The output in Athena after storing the DataFrame in hudi format.

--------------+--------------------+-----------------------+
|branch_id|             comment|         created_at|
--------------+--------------------+-----------------------+
|    13501|                    | [phone]:30:00.000|
|    14081|                    | [phone]:20:00.000|
...
--------------+--------------------+-----------------------+

Code to write the DataFrame “main_df” in Hudi format:

hudi_options = {
            'hoodie.table.name': table.name,
            'hoodie.datasource.write.recordkey.field': table.primary_key,
            'hoodie.datasource.write.partitionpath.field': partition_by,
            'hoodie.datasource.write.table.name': table.name,
            'hoodie.datasource.write.operation': "upsert",
            'hoodie.datasource.write.precombine.field': "ts_ms",
            'hoodie.upsert.shuffle.parallelism': 2,
            'hoodie.insert.shuffle.parallelism': 2
        }

main_df.write.format("hudi"). \
            options(**hudi_options). \
            mode("append"). \
            save(desturl)

The Issue is that Athena recognises int96 at timestamps and not int64 which is given by hudi. What is the fix for this?

[jackye1995]

Just FYI, a similar fix has been done after that ticket was created, and this issue should no longer exist in Athena. You can now use timestamp as the column type and there is no need to perform a conversion using something like from_unixtime(createdt/1000000).

[satishkotha]

This is a bit complicated. Hudi uses spark converters to convert dataframe type into parquet type. Spark SchemaConverters converts timestamp to int64 with logical type ‘TIMESTAMP_MICROS’.

This is because int96 is no longer supported in parquet, especially parquet-avro module. In general, int96 is discouraged going forward.

To make timestamp work, we had to

1. Change query engines to support reading parquet logical type. Example for presto. We did similar change for Hive. You probably need similar change in Athena
2. Change DLASync/HiveSync to convert logical type TIMESTAMP_MICROS as hive type ‘timestamp’. PR here

Unfortunately, there is no clean workaround. As i mentioned, this is a bit complicated. Please don’t hesitate to ping me if you have any questions.