[SUPPORT] failed to read log file

2022-02-17 17:59:40 org.apache.hudi.exception.HoodieException: Exception when reading log file at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:332) at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:178) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:103) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:96) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:291) at org.apache.hudi.table.format.FormatUtils.logScanner(FormatUtils.java:247) at org.apache.hudi.sink.bootstrap.BootstrapOperator.loadRecords(BootstrapOperator.java:232) at org.apache.hudi.sink.bootstrap.BootstrapOperator.preLoadIndexRecords(BootstrapOperator.java:139) at org.apache.hudi.sink.bootstrap.BootstrapOperator.initializeState(BootstrapOperator.java:127) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:118) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:290) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:441) at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:585) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:565) at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hudi.exception.HoodieIOException: IOException when reading logblock from log file HoodieLogFile{pathStr=‘s3a://sprs-qz-dw-prod-ue1/qz_ods/ac_account_transfer_record/20220215/.37d15126-08ab-4b3f-a45b-3a38dff0df4c_20220215212115910.log.4_0-2-1’, fileLen=0} at org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:409) at org.apache.hudi.common.table.log.HoodieLogFormatReader.next(HoodieLogFormatReader.java:120) at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:210) … 19 more Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readLong(DataInputStream.java:416) at org.apache.hudi.common.table.log.HoodieLogFileReader.isBlockCorrupt(HoodieLogFileReader.java:310) at org.apache.hudi.common.table.log.HoodieLogFileReader.readBlock(HoodieLogFileReader.java:202) at org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:407) … 21 more

Whether to catch exceptions when calculating blockSizeFromFooter？

 org.apache.hudi.common.table.log.HoodieLogFileReader#isBlockCorrupt
    long blockSizeFromFooter = inputStream.readLong() - magicBuffer.length;      // ERROR

[todd5167]

I see the file you read is an empty file, have no idea yet how this file is generated.

yes, @todd5167 from the log, it says this file is empty, maybe corrupted

Caused by: org.apache.hudi.exception.HoodieIOException: IOException when reading logblock from log file HoodieLogFile{pathStr=‘s3a://sprs-qz-dw-prod-ue1/qz_ods/ac_account_transfer_record/20220215/.37d15126-08ab-4b3f-a45b-3a38dff0df4c_20220215212115910.log.4_0-2-1’, fileLen=0}

do you think this is reproducible from your side? i meant getting the empty log file in the first place.

2022-02-17 07:14:16  104857600 .37d15126-08ab-4b3f-a45b-3a38dff0df4c_20220215212115910.log.4_0-2-1

The s3 file size is 100M, why does hudi read as an empty file. @xushiyan

[nsivabalan]

thanks!