Setting "hoodie.parquet.max.file.size" to a value >= 2 GiB leads to no data being generated

Environment: EMR cluster version 5.27 using hudi-spark-bundle-0.5.1-SNAPSHOT.jar.
“spark.executor.memory” is configured as 6018M.

Running the following code, with “hoodie.parquet.max.file.size” to 2* 1024 * 1024 * 1024 (2GiB) generates no data -

val inputDF = spark.read.format("parquet").load("s3://athena-examples-us-west-2/elb/parquet/year=2015/month=1/day=1/")

 inputDF.write 
  .format("org.apache.hudi")
  .option("hoodie.parquet.max.file.size",String.valueOf(2*1024 * 1024 * 1024))
  .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "request_ip")
  .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "request_verb")
  .option(HoodieWriteConfig.TABLE_NAME, "elb_logs_hudi_cow")
  .option(DataSourceWriteOptions.OPERATION_OPT_KEY, DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL)
  .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "request_timestamp")
  .mode(SaveMode.Overwrite)
  .save("s3://my-bucket/prefix")

All the stages and tasks complete successfully with no OOM or error messages being printed in the log. There is a commit files in .hoodie folder, but that is also empty.

Changing the option “hoodie.parquet.max.file.size” to 1024 * 1024 * 1024 (1GiB) makes everything work as expected.

[vinothchandar]

Nonetheless, bounds checking on configs needs to be improved still 😃

[vinothchandar]

Np. I did not anticipate that either 😃 . added to https://issues.apache.org/jira/browse/HUDI-89 . Closing this.