Lost Block for file

Alluxio Version: 2.1.1

Describe the bug When the file is read, no BlockInfo is found and an exception is thrown.

java.io.IOException: No BlockInfo for block(id=12700352512) of file(id=12717129727, path=/blah)
        at alluxio.client.file.FileInStream.updateStream(FileInStream.java:336)
        at alluxio.client.file.FileInStream.read(FileInStream.java:149)
        at alluxio.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:97)
        at java.io.FilterInputStream.read(FilterInputStream.java:94)

alluxio fs stat /blah produces the following exception:

2020-03-19 16:10:55,381 DEBUG BlockMasterClientServiceHandler - Enter: GetBlockInfo: blockId=33554432, options=
2020-03-19 16:10:55,381 DEBUG BlockMasterClientServiceHandler - Exit (Error): GetBlockInfo: blockId=33554432, options=
alluxio.exception.BlockInfoException: BlockMeta not found for blockId 33554432
	at alluxio.master.block.DefaultBlockMaster.lambda$getBlockInfo$1(DefaultBlockMaster.java:711)
	at java.util.Optional.orElseThrow(Optional.java:290)
	at alluxio.master.block.DefaultBlockMaster.getBlockInfo(DefaultBlockMaster.java:711)
	at alluxio.master.block.BlockMasterClientServiceHandler.lambda$getBlockInfo$0(BlockMasterClientServiceHandler.java:69)
	at alluxio.RpcUtils.callAndReturn(RpcUtils.java:113)
	at alluxio.RpcUtils.call(RpcUtils.java:78)
	at alluxio.RpcUtils.call(RpcUtils.java:53)
	at alluxio.master.block.BlockMasterClientServiceHandler.getBlockInfo(BlockMasterClientServiceHandler.java:67)
	at alluxio.grpc.BlockMasterClientServiceGrpc$MethodHandlers.invoke(BlockMasterClientServiceGrpc.java:761)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
	at 
...omitted

This lost block cannot be recovered by alluxio fs ls -Dalluxio.user.file.metadata.sync.interval=0 /blah or alluxio fs load -Dalluxio.user.file.metadata.sync.interval=0 /blah.

To Reproduce NOT CLEAR. This file was in Alluxio for a while then the block was lost.

Expected behavior The file should be able to be reloaded to recover the block.

Or there should at least be an explicit way to manually recover the file from this status.

Urgency MEDIUM

Additional context HeapBlockStore is used.

[jiacheliu3]

@ggezer Thanks so much for the investigation!

[ggezer]

Based on trace analysis this issue is due to missing this fix https://github.com/Alluxio/alluxio/pull/10901 in used version.