isLiveTable check breaks Kinesis like streams

PR https://github.com/apache/pinot/pull/7756/files introduced a check for liveTable in the path of adding new partitionGroups. This branch was introduced in setupNewPartitionGroup (prior to this PR we had just the else part)

if (isLiveTable) {
    startOffset = getPartitionGroupSmallestOffset(streamConfig, partitionGroupId);
 } else {
    startOffset = partitionGroupMetadata.getStartOffset();
 }

For new table, we go to the else, and for existing table detecting a new partitionGroup we go to if. Within the if, a call is made to getNewPartitionGroupMetadataList(streamConfig, Collections.emptyList());. For Kinesis like streams, the response from this call depends on the currentList passed. If a Kinesis like stream received empty list, it will only return the very first parent shards. This is because in Kinesis, the shards have a sequence (we started with 0, split it to 1, 2, so 1, 2 will only be returned if the current state tells it that 0 is done ingesting).

As a result of this PR change, no new shards can get detected in Kinesis.

[npawar]

Regarding, getPartitionGroupSmallestOffset(), also agree that this should be fixed regardless for Kinesis as it is used in validation manager. But that’s a much smaller and infrequent case, so i think we should prioritize first fixing the bigger case of broken new partitiongroups.

[npawar]

Yup 7743 should work as that one doesn’t use empty list for current