[SUPPORT] Hudi 0.10.1 throws NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileStatusCache.putLeafFiles(Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/FileStatus;)V

Describe the problem you faced I’m using Hudi 0.10.1 with Databricks (9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)

Trying to load a hudi data set on S3 but failed with this error NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileStatusCache.putLeafFiles(Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/FileStatus;)V

Environment Description

• Hudi version : 0.10.1

• Spark version : 3.1.2

• Hive version : N/A

• Hadoop version : N/A

• Storage (HDFS/S3/GCS…) : S3

• Running on Docker? (yes/no) : No

Stacktrace

NoSuchMethodError: org.apache.spark.sql.execution.datasources.FileStatusCache.putLeafFiles(Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/FileStatus;)V
	at org.apache.hudi.HoodieFileIndex.$anonfun$loadPartitionPathFiles$4(HoodieFileIndex.scala:604)
	at org.apache.hudi.HoodieFileIndex.$anonfun$loadPartitionPathFiles$4$adapted(HoodieFileIndex.scala:602)
	at scala.collection.immutable.Map$Map1.foreach(Map.scala:128)
	at org.apache.hudi.HoodieFileIndex.loadPartitionPathFiles(HoodieFileIndex.scala:602)
	at org.apache.hudi.HoodieFileIndex.refresh0(HoodieFileIndex.scala:360)
	at org.apache.hudi.HoodieFileIndex.<init>(HoodieFileIndex.scala:157)
	at org.apache.hudi.DefaultSource.getBaseFileOnlyView(DefaultSource.scala:199)
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:119)
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:69)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:390)

[yihua]

Here is one fix that is in progress, #7088, by optionally falling back to using Spark’s data source withHoodieROTablePathFilter (how data source read is implemented pre-0.9.0 release) instead of HoodieFileIndex, so queries on Hudi tables can work in Databricks runtime.

[makamhareesh]

hudi-spark3.1.2-bundle_2.12:0.10.1

But I believe this happens on other higher versions of Hudi bundle jars as well.