[SUPPORT] - AWS Athena snapshot query fails if there are two or more record array fields in a MoR table
See original GitHub issueDescribe the problem you faced
Querying the snapshot table (suffix -rt
) fails using Amazon Athena when the schema contains two or more record array fields.
To Reproduce
Steps to reproduce the behavior:
- Create Hudi table with two
array
columns in the the Glue data catalog (e.g see https://github.com/apache/hudi/issues/2657) - Attempt to query the
_rt
table with Athena
Environment Description
EMR 6.4.0
Athena workgroup V2 (experienced on 2021/10/20)
- Hudi version :
Tested on
0.9.0 0.8.0-amzn1
- Spark version :
3.1.2
- Hive version :
Hive 3.1.2
- Hadoop version :
Amazon 3.2.1
- Storage (HDFS/S3/GCS…) :
S3
- Running on Docker? (yes/no) :
no
Additional context
We have several columns which produce this issue, the schemas are as follows:
-
array<struct<offset:bigint,overlapping:boolean,position:string,rule_based_entity:boolean,sentiment:struct<compound:double,neg:double,neu:double,pos:double>,signal_type:string,surface_form:string,wiki_title:string>>
-
array<struct<id:string,score:string>>
Parquet tools output for this column############ Column(id) ############ name: id path: taxonomy_categories.array.id max_definition_level: 3 max_repetition_level: 1 physical_type: BYTE_ARRAY logical_type: String converted_type (legacy): UTF8 ############ Column(score) ############ name: score path: taxonomy_categories.array.score max_definition_level: 3 max_repetition_level: 1 physical_type: BYTE_ARRAY logical_type: String converted_type (legacy): UTF8
Data for this column
| [{'id': 'a3827772-bd70-4641-a017-f471fe48354e', 'score': '1'} | | {'id': 'c283b72d-2a07-4587-b2bd-ffdb4b84d06b', 'score': '0.999'} | | {'id': '0114f424-077e-4ce7-b1ea-d1a24bedee34', 'score': '0.852'} | | {'id': 'cd76217e-ae6c-41c0-9f34-d2ce082cd2e6', 'score': '0.321'}] | | [] |
This doesn’t seem to be obvious between columns, for example a column with this schema has no issues:
array<struct<end:bigint,start:bigint,text:string>>
We haven’t locked down a schema yet so we’re using spark inferred schemas on JSON source data.
The MoR table only has parquet files, no deltalogs.
Stacktrace
HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://prod-signal-hudi-experiment-datalake/hudi/documents_datalake_from_parquet_merge_on_read_upsert_v2/story_published_date=2020-01-30/cf99fa1e-a678-4dd7-a36e-72e57d50a936-0_16-34-337_20211019174019.parquet (offset=33554432, length=33554432) using org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat: Can't redefine: array
This query ran against the "pipeline_reprocessing_hudi_experiment" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: f1c60df8-e018-4210-962c-2cbb21aaa18c
Issue Analytics
- State:
- Created 2 years ago
- Reactions:1
- Comments:11 (4 by maintainers)
This seems to have been (finally) fixed in Athena engine v3 - I can no longer reproduce with my test dataset.
https://docs.aws.amazon.com/athena/latest/ug/engine-versions-reference-0003.html
For others experiencing this issue, I have just recreated this issue with Hudi 0.11.0 and the current Athena version April 22, 2022.
As expected it still needs a fix on the AWS Athena side to remedy (I haven’t checked with open source Presto)