SegmentNameGenerator: Extend interface to accept input file name

Following Slack discussion: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1624443889243500

Use case: Some tables in Pinot are used in conjunction with IdSet filtering or Lookups - and in some cases don’t have a time column.

• Hence, existing segment name generation strategies (i.e. time-based and fixed) do not allow for simple segment replacements whenever data in those “dimension” tables change (say, data for ID<X> changed)

Proposition Allow segment name generation to be based on the input file names such that segments can be named following a user provided id (in the file names).

E.g.

basedir/id1/file.parquet
basedir/id2/file.parquet

Would generate segments

<table_name>_id1.segment
<table_name>_id2.segment

Currently, the SegmentNameGenerator interface doesn’t allow input file names, therefore it is not possible to implement a strategy similar to the one presented above.

Note: If you known any other alternative to reach the use case goal, please feel free to provide ideas !

[kkrugler]

Hi @MrNeocore - yes, the scope of this issue is mapping n input files to n segments. I was pointing out that there’s one other place in the code where segments are created (SegmentProcessorFramework), but that code is (a) out of scope, and (b) doesn’t currently use the SegmentNameGenerator support in any case, though there’s a TODO comment in the code about that.

[kkrugler]

How about being able to define things via:

    "filePathPattern":".+/(.+)/.+\\.parquet",
    "segmentName":"${tableName}_${filePathPattern:\\1}.parquet"

or, for our use case:

    "filePathPattern":".+/(.+)\\.csv",
    "segmentName":"${filePathPattern:\\1}"

We could also support formatting of the input dates via say ${minTimeValue:yyyy-MM}, but maybe that’s a bridge too far.