[chart]Columns with all NA values should not be silently dropped

#8040 restored dropping columns that are all NA, but it is problematic in some visualization types. There is already #13612 for this issue in Bar Charts, for which I opened a PR, but the same applies to dual line charts if one of the lines has only null values. The error is also not informative, since it only shows the name of the column.

Expected results

The null values should be shown as nulls, without data points on the charts

Actual results

The null series is dropped by pandas.pivot_table calls which use dropna=True as default, and the user can only see the text of a KeyError:

The stacktrace for the dual line chart:

ERROR:superset.views.base:'metric1'
Traceback (most recent call last):
  File "/app/superset/views/base.py", line 181, in wraps
    return f(self, *args, **kwargs)
  File "/app/superset/utils/log.py", line 164, in wrapper
    value = f(*args, **kwargs)
  File "/app/superset/utils/cache.py", line 152, in wrapper
    return f(*args, **kwargs)
  File "/app/superset/views/utils.py", line 451, in wrapper
    return f(*args, **kwargs)
  File "/app/superset/views/core.py", line 619, in explore_json
    return self.generate_json(viz_obj, response_type)
  File "/app/superset/views/core.py", line 456, in generate_json
    payload = viz_obj.get_payload()
  File "/app/superset/viz.py", line 464, in get_payload
    payload["data"] = self.get_data(df)
  File "/app/superset/viz.py", line 1531, in get_data
    chart_data = self.to_series(df)
  File "/app/superset/viz.py", line 1500, in to_series
    ys = series[m]
KeyError: 'metric1'

How to reproduce the bug

1. Open a bar chart, a dual line chart, or other charts that use pandas.pivot_table in viz.py
2. Choose a metric that only returns null values for the selected period. This is true for e.g. sum(1)+null in most SQL dialects

Environment

I used the latest master with docker

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven’t found one similar.

[octaviancorlade]

@junlincc @kgabryje here are some screenshots and links to reproduce locally with the example data of the ones using df.pivot_table() in viz.py, only Big Number and Time-series Period Pivot seem to handle the situation currently. I am not sure if I’m missing something since the Pivot Table generates errors here, maybe the screenshots by @kgabryje are both not yet in master?

Time-series Table

Pivot Table

Big Number looks OK to me

Dual Line Chart

Time-series Period Pivot also looks OK to me

Bar Chart

Paired t-test Table

[kgabryje]

In the case of Echarts, Timeseries displays null values as zeros. Pie and Graph charts display nothing (just white background) and Box chart displays an error Error: No numeric types to aggregate