[SUPPORT] - AWS Athena snapshot query fails if there are two or more record array fields in a MoR table

Describe the problem you faced

Querying the snapshot table (suffix -rt) fails using Amazon Athena when the schema contains two or more record array fields.

To Reproduce

Steps to reproduce the behavior:

1. Create Hudi table with two array columns in the the Glue data catalog (e.g see https://github.com/apache/hudi/issues/2657)
2. Attempt to query the _rt table with Athena

Environment Description

EMR 6.4.0

Athena workgroup V2 (experienced on 2021/10/20)

• Hudi version :

Tested on

0.9.0 0.8.0-amzn1

• Spark version :

3.1.2

• Hive version :

Hive 3.1.2

• Hadoop version :

Amazon 3.2.1

• Storage (HDFS/S3/GCS…) :

S3

• Running on Docker? (yes/no) :

no

Additional context

We have several columns which produce this issue, the schemas are as follows:

• array<struct<offset:bigint,overlapping:boolean,position:string,rule_based_entity:boolean,sentiment:struct<compound:double,neg:double,neu:double,pos:double>,signal_type:string,surface_form:string,wiki_title:string>>

• array<struct<id:string,score:string>> Parquet tools output for this column

  ############ Column(id) ############
    name: id
    path: taxonomy_categories.array.id
    max_definition_level: 3
    max_repetition_level: 1
    physical_type: BYTE_ARRAY
    logical_type: String
    converted_type (legacy): UTF8
    
    ############ Column(score) ############
    name: score
    path: taxonomy_categories.array.score
    max_definition_level: 3
    max_repetition_level: 1
    physical_type: BYTE_ARRAY
    logical_type: String
    converted_type (legacy): UTF8
  

  Data for this column

    | [{'id': 'a3827772-bd70-4641-a017-f471fe48354e', 'score': '1'}      |
    |  {'id': 'c283b72d-2a07-4587-b2bd-ffdb4b84d06b', 'score': '0.999'}  |
    |  {'id': '0114f424-077e-4ce7-b1ea-d1a24bedee34', 'score': '0.852'}  |
    |  {'id': 'cd76217e-ae6c-41c0-9f34-d2ce082cd2e6', 'score': '0.321'}] |
    | []                                                                 |
  

This doesn’t seem to be obvious between columns, for example a column with this schema has no issues:

array<struct<end:bigint,start:bigint,text:string>>

We haven’t locked down a schema yet so we’re using spark inferred schemas on JSON source data.

The MoR table only has parquet files, no deltalogs.

Stacktrace

HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://prod-signal-hudi-experiment-datalake/hudi/documents_datalake_from_parquet_merge_on_read_upsert_v2/story_published_date=2020-01-30/cf99fa1e-a678-4dd7-a36e-72e57d50a936-0_16-34-337_20211019174019.parquet (offset=33554432, length=33554432) using org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat: Can't redefine: array
This query ran against the "pipeline_reprocessing_hudi_experiment" database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: f1c60df8-e018-4210-962c-2cbb21aaa18c

[Limess]

This seems to have been (finally) fixed in Athena engine v3 - I can no longer reproduce with my test dataset.

https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html

[Limess]

For others experiencing this issue, I have just recreated this issue with Hudi 0.11.0 and the current Athena version April 22, 2022.

As expected it still needs a fix on the AWS Athena side to remedy (I haven’t checked with open source Presto)