[SUPPORT] No successful commits under path

To Reproduce

Steps to reproduce the behavior:

code https://github.com/izhangzhihao/Real-time-Data-Warehouse/tree/hudi

create table

CREATE TABLE accident_claims
(
    claim_id            BIGINT,
    claim_total         DOUBLE,
    claim_total_receipt VARCHAR(50),
    claim_currency      VARCHAR(3),
    member_id           INT,
    accident_date       DATE,
    accident_type       VARCHAR(20),
    accident_detail     VARCHAR(20),
    claim_date          DATE,
    claim_status        VARCHAR(10),
    ts_created          TIMESTAMP(3),
    ts_updated          TIMESTAMP(3),
    ds                  DATE,
    PRIMARY KEY (claim_id) NOT ENFORCED
) PARTITIONED BY (ds) WITH (
  'connector'='hudi',
  'path' = '/data/dwd/accident_claims',
  'table.type' = 'MERGE_ON_READ',
  'read.streaming.enabled' = 'true',
  'write.batch.size' = '1',
  'write.task.max.size' = '1',
  'write.tasks' = '1',
  'compaction.tasks' = '1',
  'compaction.delta_seconds' = '60',
  'write.precombine.field' = 'ts_updated',
  'read.tasks' = '1',
  'read.streaming.check-interval' = '5',
  'read.streaming.start-commit' = '20210712134429',
);

insert from CDC change stream

INSERT INTO accident_claims
SELECT claim_id,
       claim_total,
       claim_total_receipt,
       claim_currency,
       member_id,
       CAST (accident_date as DATE),
       accident_type,
       accident_detail,
       CAST (claim_date as DATE),
       claim_status,
       CAST (ts_created as TIMESTAMP),
       CAST (ts_updated as TIMESTAMP),
       CAST (SUBSTRING(claim_date, 0, 9) as DATE)
FROM datasource.accident_claims;

Expected behavior

SELECT * FROM accident_claims;

should return results

But got:

Flink SQL> SELECT * FROM accident_claims;
[ERROR] Could not execute SQL statement. Reason:
org.apache.hudi.exception.HoodieException: No successful commits under path /data/dwd/accident_claims

But the sample code works:

CREATE TABLE t1(
  uuid VARCHAR(20), -- you can use 'PRIMARY KEY NOT ENFORCED' syntax to mark the field as record key
  name VARCHAR(10),
  age INT,
  ts TIMESTAMP(3),
  `partition` VARCHAR(20)
)
PARTITIONED BY (`partition`)
WITH (
  'connector' = 'hudi',
  'path' = '/data/t1',
  'write.tasks' = '1', -- default is 4 ,required more resource
  'compaction.tasks' = '1', -- default is 10 ,required more resource
  'table.type' = 'COPY_ON_WRITE', -- this creates a MERGE_ON_READ table, by default is COPY_ON_WRITE
  'read.tasks' = '1', -- default is 4 ,required more resource
  'read.streaming.enabled' = 'true',  -- this option enable the streaming read
  'read.streaming.start-commit' = '20210712134429', -- specifies the start commit instant time
  'read.streaming.check-interval' = '4' -- specifies the check interval for finding new source commits, default 60s.
);

-- insert data using values
INSERT INTO t1 VALUES
  ('id1','Danny',23,TIMESTAMP '1970-01-01 00:00:01','par1'),
  ('id2','Stephen',33,TIMESTAMP '1970-01-01 00:00:02','par1'),
  ('id3','Julian',53,TIMESTAMP '1970-01-01 00:00:03','par2'),
  ('id4','Fabian',31,TIMESTAMP '1970-01-01 00:00:04','par2'),
  ('id5','Sophia',18,TIMESTAMP '1970-01-01 00:00:05','par3'),
  ('id6','Emma',20,TIMESTAMP '1970-01-01 00:00:06','par3'),
  ('id7','Bob',44,TIMESTAMP '1970-01-01 00:00:07','par4'),
  ('id8','Han',56,TIMESTAMP '1970-01-01 00:00:08','par4');

SELECT * FROM t1;

So I didn’t get what’s wrong here…

Environment Description

• Hudi version : 0.9.0 SNAPSHOT

• Flink version : 1.12.2

• Hive version : none

• Hadoop version : 2.8.3

• Storage (HDFS/S3/GCS…) : local file system

• Running on Docker? (yes/no) : yes

Additional context

Add any other context about the problem here.

Taskmanager log: taskmanager.log.zip

[Tandoy]

If it is streaming writing, please make sure to turn on checkpoint. When the checkpoint is triggered, all data in the memory is flushed out, or a bucket has accumulated to 64MB in memory, or the total buffer size is accumulated to 256MB.

[danny0405]

Hi~How can I use checkpoint in Flink SQL client?

Take a look at this document: https://www.yuque.com/docs/share/01c98494-a980-414c-9c45-152023bf3c17?#