[SIP-6] [Embeddable Charts] Migrate visualization-specific code to js plugin

Continued from the discussion in [SIP-5] #5680

There is a decent amount of work to be done to make the visualization plugins are truly independent from the core superset.

Q1: What need to be converted to JS?

@mistercrunch : To make this frontend visualization plugin possible, we’ll need to figure out how to get rid of the visualization-specific backend code in viz.py over time, to make sure that visualizations plugins are fully defined as Javascript.

• One easy step forward is to move the backend’s query_obj code, which takes form_data as input and returns a query object (simple logic).

• The get_data method is probably harder to migrate to the front-end (takes a dataframe and returns nested objects), but it can probably be generalized quite a bit, so that it’s not visualization-specific as much as ordering pandas-like operations on the result set.

Q2: Where these new front-end methods should live?

@mistercrunch : I’m guessing you were thinking about export the adaptor and using this as an the interface between chart and the visualization plugin. Now I’m thinking we may want to export a more complex object.

export const visualization = {
  // Adapter from [SIP-5]
  adaptor, 
  // Convert formData into queryable format
  queryGenerator: (formData) => generateQuery(formData),
  // Convert formData into post-query operations 
  dataframeOperations: (formData) => (
    ['pivot_table',  { cols: [formData.cols], rows: formData.groupby }]
  ),
};

Q3: How to actually implement these two parts?

@mistercrunch : I’m not sure how to model the query vs dataframe operations, should it be incorporate into the query, or made into its own thing. Clearly on the backend we need two abstractions as one is used to interact with connectors (querying) and the other applies to all connectors (dataframe transformation).

@john-bodley : I’m also perplexed as to how to best generalize this. It seems a first step is to map arbitrary data (via a query) into a somewhat generic (potentially visualization aware) form.

@mistercrunch : I also wonder whether we could expose most of the pandas API from the frontend by doing a bit of magic, where the dataframeOperations above would somehow do something like getattr(df, command)(**props) on the backend.

[kristw]

The directory structure for each plugin will look like this. Using WordCloud as an example.

WordCloud
  |--buildQuery.js
  |--transformProps.js
  |--transformData.js
  |--metadata.js
  |
  // for the vanilla js / d3 components
  |--WordCloud.js 
  |--ReactWordCloud.jsx // convert WordCloud.js to react component
  // for the React components
  |--WordCloud.jsx
  |
  // Use temporarily until the plugin system is fully implemented
  |--adaptor.jsx
  // This will replace adaptor.jsx once the plugin system is fully implemented
  |--plugin.js
  |
  // TBD: Control configuration 
  |--controlConfig.js
  |
  // TBD: This could be class for input/output of the transformProps function
  |--formDataSchema // 
  |--dataSchema // 

What is in each file?

1. buildQuery.js

export default function buildQuery(formData) {
  return queryObject;
} 

2. transformProps.js

export default function transformProps({ 
  slice, 
  payload, 
  setControlValue,
}) {
  // Return props that are compatible with <WordCloud />
  // In the future can enforce type for the returned object if we use TypeScript. 
  return props; 
}

Note: In the future, will replace slice with specific fields, such as width, height, or formData and perhaps add type (if we use typescript) for formData

3. transformData.js

Transform props that are based on payload. The logic formerly in get_data will be moved here. For the first version, it will be called from transformProps.js, but separated into another function can be useful for future optimization.

export default function transformData({ payload, formData }) {
  return props;
}

4. metadata.js

Description of this chart

import thumbnail from './WordCloud.png';

export default new ChartMetadata({
  key: 'word-cloud',
  name: 'WordCloud',
  description: 'Lorem ipsum dolor sit amet',
  thumbnail,
});

5.1 WordCloud.js and ReactWordCloud.jsx

For vanilla components that are not React yet, we have been refactoring to make it convertible

WordCloud.js

export default function(element, props) { ... }

ReactWordCloud.jsx

import reactify from '../../utils/reactify';
export default reactify(WordCloud);

5.2 WordCloud.jsx

For components that are already React component

class WordCloud extends React.Component {
  ...
}
export default WordCloud;

6. adaptor.jsx

Shim to make it work with current Superset app. This file will be deleted after we completely move to the plugin system.

import createAdaptor from '../../utils/createAdaptor';
import WordCloud from './ReactWordCloud';
import transformProps from './transformProps';

// return function(slice, payload, setControlValue) that render the component
export default createAdaptor(WordCloud, transformProps);

7. plugin.js

This is how a chart plugin is defined. It will replace adaptor.jsx.

import buildQuery from './buildQuery';
import transformProps from './transformProps';
import metadata from './metadata';
import WordCloud from './ReactWordCloud';
export default new ChartPlugin({
  key: 'word-cloud',
  metadata,
  buildQuery,
  transformProps,
  Chart: WordCloud,
});

Plugins are installed like this

import wordCloudPlugin from './WordCloud/plugin';
wordCloudPlugin.install();

Then you can use

<SupersetDataProvider type="word-cloud" formData={...}>
  <SuperChart type="word-cloud" xxx={...} />
</SupersetDataProvider>

There is an ongoing work for plugin system in #5882 which explains what happens behind-the-scene in .install(). Please see the PR for more details.

---

@conglei

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.