Pinot query ArrayIndexOutOfBounds for some multi-value column queries

When we run the below query on our cluster via Pinot UI, we get "errorCode": 200, "message": "QueryExecutionError:\njava.lang.ArrayIndexOutOfBoundsException"

The query is simple as

select account_key from point_entry where account_key = 'xxxxx' and account_fields_values not in ('intercompany_clearing') limit 1 option(timeoutMs=10000)

Both of the columns in the query are multi-value columns.

The Server error log is below:

Caught exception while executing operator of index: 4 (query: QueryContext{_tableName='point_entry_OFFLINE', _selectExpressions=[account_key], _aliasList=[null], _filter=(account_key = 'xxxx' AND account_fields_values NOT IN ('intercompany_clearing','cash','merchant_balance','fx_clearing') AND created_at <= '1634813999'), _groupByExpressions=null, _havingFilter=null, _orderByExpressions=null, _limit=100, _offset=0, _queryOptions={responseFormat=sql, groupByMode=sql, timeoutMs=9863}, _debugOptions=null, _brokerRequest=BrokerRequest(querySource:QuerySource(tableName:point_entry_OFFLINE), pinotQuery:PinotQuery(dataSource:DataSource(tableName:point_entry_OFFLINE), selectList:[Expression(type:IDENTIFIER, identifier:Identifier(name:account_key))], filterExpression:Expression(type:FUNCTION, functionCall:Function(operator:AND, operands:[Expression(type:FUNCTION, functionCall:Function(operator:EQUALS, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:account_key)), Expression(type:LITERAL, literal:<Literal stringValue:xxxxx>)])), Expression(type:FUNCTION, functionCall:Function(operator:NOT_IN, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:account_fields_values)), Expression(type:LITERAL, literal:<Literal stringValue:intercompany_clearing>), Expression(type:LITERAL, literal:<Literal stringValue:cash>), Expression(type:LITERAL, literal:<Literal stringValue:merchant_balance>), Expression(type:LITERAL, literal:<Literal stringValue:fx_clearing>)])), Expression(type:FUNCTION, functionCall:Function(operator:LESS_THAN_OR_EQUAL, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:created_at)), Expression(type:LITERAL, literal:<Literal stringValue:1634813999>)]))])), orderByList:[], limit:100, queryOptions:{responseFormat=sql, groupByMode=sql, timeoutMs=9863}))})

[dongxiaoman]

For now we have pin pointed to a segment that was generated long ago. Could be the data that was wrongly generated; for now we can work around this by adding a time filter to skip those segments. Will keep digging

[dongxiaoman]

Forgot to close it. Happens only to one segment that is probably generated by some version of library that may have issues.