[SUPPORT] Got NoSuchElementException while using hudi 0.10.0 and Flink (COW)

Tips before filing an issue

• Have you gone through our FAQs? Yes

• Join the mailing list to engage in conversations and get faster support at [e-mail].

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced While using hudi 0.10.0 with Flink writing data to S3, sometimes got error like below

Caused by: java.util.NoSuchElementException: No value present in Option
        at org.apache.hudi.common.util.Option.get(Option.java:88)
        at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:115)
        at org.apache.hudi.io.FlinkMergeHandle.<init>(FlinkMergeHandle.java:70)
        at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:497)
        at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:143)
        at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:183)
        at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:460)
        at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
        at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:453)
        at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:130)
        at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:150)
        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
        at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:89)
        at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:218)

https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L115 Seems the fileId was not present in the partition

here is my configs

CREATE TABLE data1 (
  schema STRING,
  type STRING,
  uuid STRING,
  `time` STRING,
  page STRING,
  payload STRING,
  dt STRING
)
PARTITIONED BY (`dt`)
WITH (
  'connector' = 'hudi',
  'path' = 's3a://xxxx/data',
  'hoodie.embed.timeline.server' = 'true',
  'write.precombine.field' = 'time',
  'hoodie.parquet.max.file.size' = '62914560',
  'index.bootstrap.enabled' = 'true',
  'hoodie.parquet.block.size' = '62914560',
  'hoodie.metadata.enable' = 'false',
  'hoodie.datasource.write.recordkey.field' = 'uuid',
  'write.tasks' = '4',
  'hoodie.datasource.write.hive_style_partitioning' = 'true',
  'index.state.ttl' = '1.5D',
  'write.bucket_assign.tasks' = '1',
  'read.streaming.enabled' = 'false',
  'table.type' = 'COPY_ON_WRITE',
  'index.global.enabled' = 'false',
)

To Reproduce

Steps to reproduce the behavior: It was hard to reproduce since it happens every serval days

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version : 0.10.0

• Spark version :

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS…) : S3

• Running on Docker? (yes/no) : No

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[danny0405]

Have fixed it in https://github.com/apache/hudi/pull/5917

[shqiprimbkodelabs]

The same issue is happening to me:

Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 9 for operator stream_write: HUDI_POINT_TIME_SERIES (2/4)#1. Failure reason: Checkpoint was declined. at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186) at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605) at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315) at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258) ... 22 more Caused by: java.util.NoSuchElementException: No value present in Option at org.apache.hudi.common.util.Option.get(Option.java:89) at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:120) at org.apache.hudi.io.FlinkMergeHandle.<init>(FlinkMergeHandle.java:70) at org.apache.hudi.io.FlinkConcatHandle.<init>(FlinkConcatHandle.java:53) at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:489) at org.apache.hudi.client.HoodieFlinkWriteClient.insert(HoodieFlinkWriteClient.java:170) at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$0(StreamWriteFunction.java:181) at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461) at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608) at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454) at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131) at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219) ... 33 more