Support for automatic index and timezone recovery from Parquet files.

When Parquet files are created by Wrangler/Pandas some metadata are stored into the file with hints about how to reconstruct Indexes correctly or how to localize a datetime column with a specific time zone.

• When wr.s3.to_parquet(index=True) we want to preserve the current behaviour of materialize the index(es) as a parquet column. This is important because non materialised index(es) will not be detected by engines such Spark, PrestoDB, Redshift Spectrum, Athena, Hive, etc.
• Support for index(es) recovery following Pandas metadata injected in the files.
• Support for MultiIndex
• Support for RangeIndex
• Support for datetime timezone recovery.

Related commits:

• Add support for timezone and index for wr.s3.read_parquet() https://github.com/awslabs/aws-data-wrangler/commit/0434ec72c8096bc7112d1cda10b7c551080a1f6b
• Improve index recovery https://github.com/awslabs/aws-data-wrangler/commit/50b80aee13c5c53db644faf284d89aedcc6ab6e7
• Add support for multiindex recovery https://github.com/awslabs/aws-data-wrangler/commit/c64e7e0ccefeaa4814872d31a5ad8f7a7b3f3e69

Original discussion: https://github.com/awslabs/aws-data-wrangler/pull/339

cc: @alexifm @Digma

[alexifm]

Hey, I think this is awesome! I tested it out a bunch with different index types and partitions and I think the only round trip issues I found are known pyarrow issues, mostly relating to categoricals. Specific to what the wrangler is responsible for, I can’t find any issues.

I did not test anything with the timezones.

[Digma]

Tested it and everything seems to work ok for us. Thanks again!