Time Series viz shows incorrect legend for multiple group by columns

If you use multiple group by columns in the time series viz, the legend shows all permutations of the values rather than just the ones that exist in the data. This is a regression.

Expected results

The legend should only show values for the series in the chart.

Actual results

The legend shows all permutations of column values that are in the chart.

Screenshots

The legend incorrectly shows all permutations of 0 and 1, but the only actual series are (0,0) and (1,1).

How to reproduce the bug

1. Run a query like: SELECT 0 as a, 0 as b, '2021-07-28' as ds UNION SELECT 1 as a, 1 as b, '2021-07-28' as ds
2. Explore chart. Edit dataset to make ds temporal.
3. Switch to time series viz. Use metric count(*), group by a and b. Turn on legend.
4. View incorrect legend.

Environment

(please complete the following information):

• superset version: superset version up to date with master as of 2021-07-28
• python version: python --version 3.8.11
• node.js version: node -v v14.16.1

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven’t found one similar.

Additional context

We just started observing this issue in this week’s release, so this regression likely occurred sometime during the last ~10 days.

Unsure whether this was caused by a change in superset-frontend or superset-ui. @zhaoyongjie any thoughts?

[zhaoyongjie]

@serenajiang Thanks for reporting this issue, I file a PR to fix that.

[zhaoyongjie]

It’s a Pandas pivot_table limitations. https://github.com/pandas-dev/pandas/issues/18030

After filling other value, the pivot_table is unable to do aggregation.