[FIXED] Presto cannot query hudi table
See original GitHub issueDescribe the problem you faced
I made a non-partitioned Hudi table using Spark. I was able to query it with Spark & Hive, but when I tried querying it with Presto, I received the error Could not find partitionDepth in partition metafile
.
To Reproduce
Steps to reproduce the behavior:
- Use an an emr-5.28.0 cluster
- Run spark shell:
spark-shell --packages org.apache.hudi:hudi-spark-bundle_2.11:0.5.1-incubating,org.apache.spark:spark-avro_2.11:2.4.4 \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --deploy-mode client
- Run spark code:
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.hudi.hive._
import org.apache.hudi.keygen.NonpartitionedKeyGenerator
val inputPath = "s3://path/to/a/parquet/file"
val tableName = "my_test_table"
val basePath = "s3://test-bucket/my_test_table"
val inputDf = spark.read.parquet(inputPath)
val hudiOptions = Map[String,String](
RECORDKEY_FIELD_OPT_KEY -> "dim_advertiser_id",
PRECOMBINE_FIELD_OPT_KEY -> "update_time",
TABLE_NAME -> tableName,
KEYGENERATOR_CLASS_OPT_KEY -> classOf[NonpartitionedKeyGenerator].getCanonicalName, //needed for non partitioned table
HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY -> classOf[NonPartitionedExtractor].getCanonicalName, //needed for non partitioned table
OPERATION_OPT_KEY -> BULK_INSERT_OPERATION_OPT_VAL,
HIVE_SYNC_ENABLED_OPT_KEY -> "true",
HIVE_TABLE_OPT_KEY -> tableName,
TABLE_TYPE_OPT_KEY -> COW_TABLE_TYPE_OPT_VAL,
"hoodie.bulkinsert.shuffle.parallelism" -> "10")
inputDf.write.format("org.apache.hudi").
options(bulk_insert_hudiOptions).
mode(Overwrite).
save(basePath);
- Querying the table in Spark or Hive both work
- Querying the table in Presto fails
[hadoop@ip-172-31-128-118 ~]$ presto-cli --catalog hive --schema default
presto:default> select count(*) from my_test_table;
Query 20200211_185123_00018_pruwt, FAILED, 1 node
Splits: 17 total, 0 done (0.00%)
0:02 [0 rows, 0B] [0 rows/s, 0B/s]
Query 20200211_185123_00018_pruwt failed: Could not find partitionDepth in partition metafile
com.facebook.presto.spi.PrestoException: Could not find partitionDepth in partition metafile
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:200)
at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Could not find partitionDepth in partition metafile
at org.apache.hudi.common.model.HoodiePartitionMetadata.getPartitionDepth(HoodiePartitionMetadata.java:75)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.getTableMetaClient(HoodieParquetInputFormat.java:209)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.groupFileStatus(HoodieParquetInputFormat.java:158)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:69)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:288)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:371)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:264)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:96)
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:193)
... 7 more
Expected behavior
Presto should return a count of all the rows. Other Presto queries should succeed.
Environment Description
-
EMR version: emr-5.28.0
-
Hudi version : 0.5.1-incubating, 0.5.0-incubating
-
Spark version : 2.4.4
-
Hive version : 2.3.6
-
Hadoop version : 2.8.5
-
Presto version: 0.227
-
Storage (HDFS/S3/GCS…) : S3
-
Running on Docker? (yes/no) : no
Stacktrace
Included in “Steps to reproduce”.
Additional Info
When I used one of the columns as a partition column, I was able to query the table in Spark using spark.read.format("org.apache.hudi").load(basePath + "/*")
. However, querying it in Hive resulted in:
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1580774559033_0082_2_00, diagnostics=[Vertex vertex_1580774559033_0082_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: my_test_table initializer failed, vertex=vertex_1580774559033_0082_2_00 [Map 1], java.lang.NullPointerException
at org.apache.hudi.hadoop.HoodieHiveUtil.getNthParent(HoodieHiveUtil.java:66)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.getTableMetaClient(HoodieParquetInputFormat.java:313)
at org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:98)
at org.apache.hudi.hadoop.InputPathHandler.<init>(InputPathHandler.java:58)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:71)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:288)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:442)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:561)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]
Vertex killed, vertexName=Reducer 2, vertexId=vertex_1580774559033_0082_2_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1580774559033_0082_2_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1580774559033_0082_2_00, diagnostics=[Vertex vertex_1580774559033_0082_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: my_test_table initializer failed, vertex=vertex_1580774559033_0082_2_00 [Map 1], java.lang.NullPointerException
at org.apache.hudi.hadoop.HoodieHiveUtil.getNthParent(HoodieHiveUtil.java:66)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.getTableMetaClient(HoodieParquetInputFormat.java:313)
at org.apache.hudi.hadoop.InputPathHandler.parseInputPaths(InputPathHandler.java:98)
at org.apache.hudi.hadoop.InputPathHandler.<init>(InputPathHandler.java:58)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.listStatus(HoodieParquetInputFormat.java:71)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:288)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:442)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:561)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1580774559033_0082_2_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1580774559033_0082_2_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
Querying it in presto-cli returned 0 rows.
Issue Analytics
- State:
- Created 4 years ago
- Comments:6 (3 by maintainers)
I found the problem. We had client-side encryption configured for Spark & Hive using EMRFS, but not for Presto.
Update: This problem does not occur in the docker environment. In the docker demo env, I was able to create a non-partitioned table in Spark (saved to hdfs), use run_sync_tool.sh to sync it to hive, and then query it successfully from presto. (It still made the .hoodie_partition_metadata file though).