[SIP-43] Unified Chart Controls

Disclaimer: regardless of the details embedded, this document is a conceptual proposal intended to start conversations. We are not committed to implement all or any of the proposed changes yet.

[SIP-43] Unified Chart Controls

Motivation

After obtaining tabular data from datasources, visualizations need to know how to map data columns to marks and channels in visualizations. Sometimes the visualization may also require or the users would like to perform additional transformations before plotting (compute moving average, pivot, aggregate, transpose, sample, window, etc.), which are not possible/convenient to do at the datasource/SQL query level.

The concept of post-processing is not new to Superset. Historically, we had some flavors of these known as advanced analytics in a few charts such as the line chart, but they lack clarity and consistency.

Consistency: Most of these transformations are implemented in viz.py, with each visualization having its own non-standardized post-processor. This Python module has grown out of control and become difficult to maintain. The coupling between visualization and query response makes it difficult to improve some important features such as chart slice caching, embeddable charts and visualization plugins.

We have been slowly migrating to a new visualization-independent query API (SIP-5, SIP-6, #6220). The idea is to decouple data querying with visualizations, move most visualization-specific transformations to the frontend. The new API will always return tabular data. It is up to each visualization plugin to take the output, optionally run it through a generalized post-processing API on the server side before (#9427), and pass the tabular data to the visualization code, which handle the rest of the visualization-specific transformation on the client-side.

Clarity: Some post-processing can be implicit but straightforward, e.g., when users want to add moving averages to a line chart, we simply introduce a form control that computes a derived column. It is easy to infer how that column should be presented in the visualization. However, while there is room for abstraction, this approach requires developers to create custom controls for every visualization, each with their own transformProps logics. Like many basic chart controls in Superset, these controls affect both data manipulation and presentation, so they still unavoidably bind data querying to visualizations. It may be straightforward to use them in one simple chart, but it quickly becomes confusing when there are a lot of different controls across many visualizations—both developers and users sometimes have to guess what’s happening under the hood.

Flexibility: In addition to lack of clarity and consistency, the custom control approach also fall short of supporting more powerful visualizations. For example, following table chart with mock data is common in top-line business reports. It compactly displays multiple metrics (bookings and revenue) across multiple dimensions (state, user type) and multiple time periods (point-in-time measurement and 7 day moving average):

Suppose each metric and dimension is a column in the database, and each row is their values at a given date. Currently, in order to create this output table, users have to write very complex SQL queries and use a virtual datasource. But it’s actually possible to very quickly build the same chart using a combination of Pandas post-processing operators, without writing complex database queries.

Proposed Change

To solve the challenges above, we propose to (1) add a Transform section for server-side post-processing and (2) rearrange the Customize controls in the control panel.

The Transform Section

In the Transform section, users can specify stackable atomic transform operators mapped directly to the pandas_postprocessing API already implemented in the backend. For example, the screenshot below shows the controls popup for Rolling Window transformation:

This one-on-one mapping between transform controls and post processing operators makes it easy for documentation. We can just point users to the Python API spec for pandas_postprocessing.

After applying a transformation, users should be able to view the intermediate and final tabular data (the final results returned by the server will always be tabular). We can add a button to switch between data view and chart view:

Or implement the split-view proposed in SIP-34.

Clicking on the accordion list items in Transform will switch between the intermediate transformed results.

The Customize Section

Superset introduced the Data vs Customize tab to reduce clutter in the control panel. This is helpful for charts with many options. But it also makes the Customize options difficult to discover. A lot of users don’t even know they can add pagination to the table chart.

To simplify the user experience, we intend to move the Customize tab to a new Customize section under the main tab and add a Columns section before other chart rendering options.

The Columns section configures per-column meta data such as d3 format, tooltip template, suffix/prefix, and conditional formatting, corresponding to the final data output. Not all visualizations have customizable rendering options, but those with rendering options will always have the Columns section before other rendering options. We believe with refined UI hierarchy, it’s possible to resurface the customize controls without creating clutters.

The full mockup can be found here.

Long-term Plan

In the future, all visualization controls will follow three simple steps: Query (datasource queries) → Transform (server side post processing) → Customize (chart rendering). This is akin to the visualization grammar used by Vega and Vega-lite. This separation of concerns and unification of control semantics make it super clear what each control is responsible for.

By moving column mapping and chart rendering logics to Customize, we can also remove control overrides in the Query section—currently there are too many variants of metrics, columns, and groupby fields querying the same thing but are stored differently (apache-superset/superset-ui#485 provided a mask to help developers; end users may still be confused). This not only greatly simplifies the code, but also makes it easier to switch between visualization types—all control values for Query, Transform, and even Columns can be easily retained.

In the future (beyond the scope of this SIP), if it’s too tiresome to edit all the transformations one by one, there are many ways to simply the Transform section for users:

1. We can hide complex operations behind custom operators that are either a preset of other operators, or an arbitrary Python function running in sandbox.
2. We can add a switch of Simple v.s. Advanced mode. The Advanced mode is what described above. In the Simple mode, users specify transformations using less controls, similar to Advanced Analytics. Each control could potentially represent one or multiple transformations with reasonable defaults.

New or Changed Public Interfaces

• New React components for the Transform controls and popup modals, which will interact with the post_processing field in the new API (/api/v1/chart/data).
• New shared controls for the Columns section.
• New design pattern for control panel to optimize the hierarchy of existing controls (data source, time, etc.).
• Refactor control panel config registrations and chart control overrides.

New dependencies

No new NPM/Pip dependencies needed.

Migration Plan and Compatibility

This change has profound implications on the way we think of charting in Superset. In addition to exposing post-processing API, it also proposes changes to the Query controls.

We will start with implementing the controls for one example visualization type (the table chart for example), then migrate others one by one. For each visualization type, we have to do the following:

• Step 1: migrate to the new /chart/data API
• Step 2: add Transform controls, implement the data vs chart split view
• Step 3: move Customize controls to the main tab, add Columns controls
• Step 4: rename and simplify Query controls, db migration may be needed

Rejected Alternatives

• Add yet another custom control or visualization type for the advanced table chart: the use case is too specific. There are too many operations in the underlying transformation, it becomes too opaque to the users.
• Only add the transform section to charts which need it: this does not solve the long-standing problem of inconsistent and speculative chart controls, but still introduces a new pattern for charting.

References

• [SIP-34] A new design direction, system, and process for Superset
• Full mockup for the unified controlled based on current UI

[ktmud]

I feel the SIP template could use a “Technical Challenges and Risks” section.

[rusackas]

Since this issue is closed, we’ll also consider the SIP/proposal process closed as well. If you want to rekindle this proposal, please re-open this Issue, and send a new [DISCUSS] thread to the dev@ mailing list. Thank you!