Star-tree Optimizations

Problems:

1. Certain predicates (e.g. NOT_EQUAL) can select too many branches, limiting the performance of star-tree
2. Star-tree does not support filters with OR

Potential solutions for 1:

• During aggregation, negate the aggregation result of the non-selected branches from the star node branch. This solution could become complicated for multiple predicates.
• Create a derived column for the predicate result, and apply star-tree on top of it. This solution can only work on the fixed predicate.
• First use inverted index to do the filtering, then use pre-aggregated docs to cover the contiguous selected docs

Potential solution for 2:

• Use bitmap to store the selected docs for each predicate under OR, then union the bitmaps

[Jackie-Jiang]

You may read the star-tree doc to get a high level idea of how star-tree works: https://docs.pinot.apache.org/basics/indexing/star-tree-index?q=star+tree+index

[atris]

@jpohanka Actually, that would require fundamental changes to the structure of the star tree index, since the current structure would lead to a double counting of certain values for multiple dimensions.

Let me talk to @Jackie-Jiang and get back