Adding batch operation APIs for PinotFS

Currently PinotFS doesn’t support batch operation API extensions easily.

For some usage such as SegmentDeletionManager. once has to iterate over all files checking for deletion and then do the actual delete. This doesn’t seem to be an issue with localFS, but for many of the cloud FS, there’s more efficient batch APIs to reduce significantly on the remote request overheads.

Propose to

1. add delete / copy / move (List<URI> segmentUris, ...) API to its single URI variant.
2. make default implementation for them to fall back to looping each individually.

[walterddr]

for this issue it is more the multiple-round-trip per segment when we operate on the entire table.

Even if the deleteSegment does the deletion only without the move. on a large enough table it will still take 10s of minutes to complete the deletion since it requires pinot to issue delete(URI segmentUri) sequentially segment after segment instead of leveraging the underlying PinotFS impl, which might have a much more efficient way to batch operator on a list of segment URIs

[Jackie-Jiang]

@mcvsubbu Which config are you referring to? I find a config for the retention days before deleting the segment from the deleted segment dir, but didn’t find one to skip moving the segment to the deleted segment dir. In SegmentDeletionManager.removeSegmentFromStore(), segment is always moved.