[QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?

when I use spark sql create table and set hoodie.datasource.write.operation=upsert. delete sql （like pr #5215 ）, insert overwrite sql etc will still use hoodie.datasource.write.operation to update record, not delete, insert_overwrite etc.

eg: create a table and set hoodie.datasource.write.operation upsert when I use sql to delete, the delete operation key will be overwrite by hoodie.datasource.write.operation from table or env, OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL will not effect, overwrite to upsert

withSparkConf(sparkSession, hoodieCatalogTable.catalogProperties) {
  Map(
    "path" -> path,
    RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
    TBL_NAME.key -> tableConfig.getTableName,
    HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
    URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
    KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
    SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
    OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL,
    PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
    HiveSyncConfig.HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
    HiveSyncConfig.HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
    HoodieWriteConfig.DELETE_PARALLELISM_VALUE.key -> "200",
    SqlKeyGenerator.PARTITION_SCHEMA -> partitionSchema.toDDL
  )
} 

so, when use sql, what about don’t write it to hoodie.properties, confine it when sql check, command generated itself in runtime.

[dongkelun]

Not only hoodie.datasource.write.operation, but also other properties such as hoodie.table.name has the same problem. I think we should find all the parameters with similar problems

[yihua]

We’re going to track the fix in HUDI-4001. Closing this issue.