[SUPPORT] How to use hudi-defaults.conf with Glue

Describe the problem you faced

I tried to use Hudi hudi-defaults.conf with Glue and tried to set the path of the file using Spark Config and Python Environment config and it doesn’t work. I checked this issue https://github.com/apache/hudi/pull/4167 but i can’t find a clear idea about how to use it.

Spark Config: pyspark

spark = SparkSession.builder.config('spark.serializer','org.apache.spark.serializer.KryoSerializer')
.config('spark.sql.hive.convertMetastoreParquet','false')
.config('spark.yarn.appMasterEnv.HUDI_CONF_DIR',args['HUDI_CONF_DIR'])
.config('spark.executorEnv.HUDI_CONF_DIR',args['HUDI_CONF_DIR']).getOrCreate()

Env Config:

HUDI_CONF_DIR='s3://glue-development-bucket/scripts/hudi-conf/hudi-default.conf'
os.environ['HUDI_CONF_DIR'] = args['HUDI_CONF_DIR']

I am getting the same error every time, I am not sure if there is a clear example about how to use this feature with spark or Glue.

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version : 0.10.1

• Spark version : 3.1.1

• Hive version : 2.3.7

• Storage (HDFS/S3/GCS…) : S3

• Running on Docker? (yes/no) : no

Add the stacktrace of the error.

2022-04-19 00:34:37,012 WARN [Thread-10] config.DFSPropertiesConfiguration (DFSPropertiesConfiguration.java:getConfPathFromEnv(188)): Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
--
2022-04-19 00:34:37,085 WARN [Thread-10] config.DFSPropertiesConfiguration (DFSPropertiesConfiguration.java:addPropsFromFile(131)): Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file

In the test example, it uses DFSPropertiesConfiguration.refreshGlobalProps(); to refresh, but i am not sure how to use this with pyspark config.

[moustafaalaa]

The warning disappeared, I will verify it is working fine and share the output. It still there. I added the details in the next comment

[nsivabalan]

closing it out since the linked PR is landed. thanks!