[SUPPORT] Caused by: java.lang.IllegalArgumentException: ALREADY_ACQUIRED

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

1.flink sql create table

CREATE TABLE if not exists test_hudi_flink_mor_occ (
  uuid VARCHAR(200) PRIMARY KEY NOT ENFORCED,
  name VARCHAR(100),
  age INT,
  ts TIMESTAMP(0),
  `partition` VARCHAR(20)
)
PARTITIONED BY (`partition`
)
WITH (
  'connector' = 'hudi',
  'path' = '/project/ccr_ai_upgrade/ccr_ai_upgrade/aiui/flink/demo/test_hudi_flink_mor_occ',
  'table.type' = 'MERGE_ON_READ',
  'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.ComplexAvroKeyGenerator',
  'hoodie.datasource.write.recordkey.field' = 'uuid',
  'write.precombine.field' = 'ts',
  'hoodie.datasource.write.hive_style_partitioning' = 'true',
  'hoodie.write.concurrency.mode' = 'optimistic_concurrency_control',
  'hoodie.cleaner.policy.failed.writes' = 'LAZY',
  'hoodie.write.lock.provider' = 'org.apache.hudi.hive.transaction.lock.HiveMetastoreBasedLockProvider',
  'hoodie.write.lock.hivemetastore.database' = 'hive',
  'hoodie.write.lock.hivemetastore.table' = 'test_hudi_flink_mor_occ_lock'
)

2.flink sql insert into

insert into test_hudi_flink_mor_occ values ('a','liangji',18,TIMESTAMP '2022-11-20 18:00:00','2022-11-20')

Environment Description

• Hudi version :0.12.1

• Flink version :1.14.0

• Hadoop version :2.6.0

Stacktrace

org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'stream_write: test_hudi_flink_mor_occ' (operator 905882b372467a732faa32d0b002df8d).
	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:545)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$start$0(StreamWriteOperatorCoordinator.java:187)
	at org.apache.hudi.sink.utils.NonThrownExecutor.handleException(NonThrownExecutor.java:146)
	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:133)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Executor executes action [handle end input event for instant 20221128171431329] error
	... 9 more
Caused by: java.lang.IllegalArgumentException: ALREADY_ACQUIRED
	at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
	at org.apache.hudi.hive.transaction.lock.HiveMetastoreBasedLockProvider.acquireLock(HiveMetastoreBasedLockProvider.java:137)
	at org.apache.hudi.hive.transaction.lock.HiveMetastoreBasedLockProvider.tryLock(HiveMetastoreBasedLockProvider.java:113)
	at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:73)
	at org.apache.hudi.client.HoodieFlinkWriteClient.writeTableMetadata(HoodieFlinkWriteClient.java:280)
	at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:270)
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:235)
	at org.apache.hudi.client.HoodieFlinkWriteClient.commit(HoodieFlinkWriteClient.java:125)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doCommit(StreamWriteOperatorCoordinator.java:530)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:506)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.commitInstant(StreamWriteOperatorCoordinator.java:477)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:420)
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:278)
	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130)

[danny0405]

Yeah, seems a bug, seems all the invocations of #writeTableMetadata are already guarded by the txn lock. We can remove the lock in #writeTableMetadata i think, please fire a JIRA and fix it if you are interested.

[nsivabalan]

thanks!