[SIP-82] - Improving Superset Theming

[SIP-82] - Improving Superset Theming

Motivation

Recently we did a POC with the objective of changing the colors of the Superset theme to analyze how it affected the application. The reason was to understand how far we are from allowing Superset users to customize the theme. The objective of this SIP is to document the findings of the POC and to propose some efforts to improve Superset theming capabilities. We understand that some of the efforts may need additional SIPs or are linked with existing SIPs. It’s not the objective of this SIP to dive into the details of all the efforts but to propose an overview of the work necessary and to propose solutions to some of them.

The Superset theme configuration has the following structure:

const defaultTheme = {
  borderRadius: 4,
  colors: {
    text: {...},
    primary: {...},
    secondary: {...},
    grayscale: {...},
    error: {...},
    warning: {...},
    alert: {...},
    success: {...},
    info: {...},
  },
  opacity: {...},
  typography: {
    families: {...},
    weights: {...},
    sizes: {...},
  },
  zIndex: {...},
  transitionTiming: 0.3,
  gridUnit: 4,
};

We started by changing the theme’s primary, secondary, and grayscale colors to green, purple, and gold respectively using colors from Ant Design color palettes.

Primary	Secondary	Grayscale

We also changed text colors to magenta. Here’s the result of these changes on some of our main screens:

Some interesting conclusions can be drawn from the above images:

• Some areas of the application are not affected at all by theme changes like the main header, backgrounds, links, and charts
• Some components are partially affected by the theme changes like the Select
• There’s a clear difference in how the background colors are handled between SQL Lab and the other modules
• The secondary color is rarely seen
• Changing the text color to magenta didn’t produce any effect
• Many pages in Settings are rendered on the server-side and are not connected with the theme at all

Proposed Change

To deal with these problems and make Superset themeable, we need to complete some efforts.

Standardize theme access

If we do a quick search for “supersetTheme" we can see that the theme object is accessed directly in some files. This means that these files will not be affected by potential changes to theming properties. At the time of writing, we have 40 files referencing the theme directly. We need to fix these files and standardize theme access.

Get rid of the remaining LESS files

We need to complete our work and remove the remaining LESS files from the application. At the time of writing, we have 17 LESS files remaining. Given that each LESS file contains many style definitions that may or may not be in use by the application, this task will require a great deal of effort and can potentially introduce regressions.

Get rid of the remaining CSS files

Some plugins are using CSS files to configure the theme properties. We need to migrate them to Emotion and make the necessary adjustments to colors, fonts, etc. At the time of writing, we have about 20 CSS files.

Remove hard-coded typography and colors

Many components are still using hard-coded typography (font families, weights, sizes) and colors. At the time of writing, we don’t have an estimate on how many components need to be fixed.

Write linting rules to find and warn on non-themed styles we care about in Emotion

One effort that can help us find non-themed styles and prevent future incompatibilities is to write linting rules that are capable of finding these errors. It is worth spending some time trying to make this work because it can really buy us time in the short and long term.

Change the way we import Ant Design from LESS to CSS

Right now we are importing Ant Design LESS files directly to override some of its variables. As we progress with removing LESS dependencies we also need to change the way we import Ant Design to use the CSS version as described in the installation docs.

Migrate server-side pages to the client-side

Many Superset pages are still being rendered on the server-side. Some examples include:

• Users list
• Roles list
• Role Level Security filters
• Action Log list
• Info page

Right now these pages aren’t affected by theming changes. We considered maintaining these pages on the server-side and try to apply the theme configurations to them. After analyzing the complexities and limitations of the solution, we recommend migrating them to the client-side since this is also a requirement for the single-page application (SPA) effort. As part of this work, we may need new APIs to support/hydrate these pages.

Improve the theme structure definition

The theme structure definition constitutes an API between Superset and theme providers. A poorly defined structure will make it difficult for theme providers to understand which parts of the application are affected by the changes, may limit the areas for customization, and can lead to unnecessary migration while evolving.

Looking at the structure of our current theme, we found that we needed to make some improvements to support multiple themes. To come up with the improvement suggestions we took inspiration from the following references during the research phase:

• Material UI theme customization tool
• Material Design - Color system
• Theme UI Spec

To support multiple themes, we are suggesting the following changes to the colors theme structure:

colors: {
  background: {
    default: <value>,
    surface: <value>,
    popover: <value>,
  },
  text: {
    label: <value>, // remove
    help: <value>, // remove
    default: <value>,
    hint: <value>,
    disabled: <value>,
  },
  primary: {},
  secondary: {}, // remove
  grayscale: {},
  error: {},
  warning: {}, // make this yellow
  alert: {}, // remove
  success: {},
  info: {},
},

Create a background category with the default, surface, and popover subcategories

We noticed that we need these categories to support the customization of background structures like dashboard cards, popovers, tooltips, navigation panels, etc. These colors are independent of the primary and grayscale categories. This became really clear when simulating a dark theme.

Modify the text category

We need to add more text categories to reflect the different use cases found in the application. The default category is used to display the majority of texts in the application. We also created the hint and disabled categories and removed the label and help categories.

To deal with texts that appear on top of primary and grayscale colors we’ll use algorithms to automatically determine the most appropriate color.

Remove the secondary color

Currently, the secondary color is rarely seen in Superset. We’re proposing to adjust these rare places to use primary and grayscale colors and remove the secondary category, simplifying the theme configuration.

Merge the warning and alert categories

We can simplify the configuration of our functional colors by merging the warning and alert categories. The idea is to keep the warning category with the colors from the alert category because both the name and the gold colors are most commonly found in other applications and align better with the users’ basic understanding of color functions.

We can also improve typography definitions. Although we have families, weights, and sizes, it’s not always clear where they are being used and how they are being compounded to create common text elements. Because the composition is not clearly established, developers tend to mix different weights and sizes for components that should belong to the same typography category.

To improve the standardization of typography categories and at the same time maintain the flexibility of the atomic configurations, we propose to enhance our theme with the introduction of presets. Presets are a combination of families, weights, sizes, and line heights that form the typography building blocks of the application. They are not arbitrarily defined but follow the design system specification which means that any change to them should be a reflection of a change in the design system. One important step we can take to make sure this statement remains true in the future is to explicitly set code owners for the theme files with members of the design team and selected engineers.

Here’s the typography building blocks as defined by the design system (@kasiazjc):

The proposed changes are to adjust the weights and sizes categories to reflect the design system and to create the presets category:

typography: {
  families: {
    sansSerif: `'Inter', Helvetica, Arial`,
    serif: `Georgia, 'Times New Roman', Times, serif`,
    monospace: `'Fira Code', 'Courier New', monospace`,
  },
  weights: {
    light: 200, // remove
    regular: 400, // normal -> regular
    medium: 500,
    semiBold: 600,
    bold: 700, // 600 -> 700
  },
  sizes: {
    xxs: 9,  // remove
    xs: 10, // remove
    s: 12,
    m: 14,
    l: 16,
    xl: 18, // 21px -> 18px
    xxl: 24, // 28px -> 24px
  },
  presets: {
     heading1: {
        fontFamily: families.sansSerif,
        fontWeight: weights.medium,
        fontSize: sizes.xxl,
        lineHeight: 1.3,
     },
     bodyDefault: {
        fontFamily: families.sansSerif,
        fontWeight: weights.normal,
        fontSize: sizes.m,
        lineHeight: 1.3,
     },
     monospace: {
        fontFamily: families.monospace,
        fontWeight: weights.normal,
        fontSize: sizes.s,
        lineHeight: 1.3,
     },
     ... // remaining presets
  }
}

With these definitions, we can easily map them to the screens and maintain a consistent design throughout the application:

The suggested changes here constitute the first draft for the theme categories and are not complete. As this is an iterative approach, we’re open to further modifications to the theme structure while working on theming tasks. We can use the dark theme implementation as the compass to define if we need additional categories or if further simplifications can be made.

Define a way to apply our theme to Ant Design’s theme

One problem we currently have is how to customize Ant Design’s theme directly. Right now we have many components that override the native styles by changing Ant Design CSS properties manually. Some components are extensions to the native components and add new features or modify the way a component behaves. Others are simply an extra layer to change CSS. The problem with that approach is that it increases the probability of breaking changes when upgrading Ant Design versions. One of the bigger problems right now is that Ant Design uses LESS for defining theme variables and the support for dynamically changing these variables is really poor. Luckily, after many requests from the community, the Ant Design team finally decided to support CSS-in-JS in Ant Design 5. This will probably change the way we are modifying native styles and reduce the number of custom components on our end. We don’t have an estimate for this version yet so we need to come up with a temporary solution. During the POC we come up with three different possibilities:

Upgrade Ant Design to the latest version and use CSS variables

One possibility is to use the new CSS variables feature available in the latest version of Ant Design. This feature allows us to dynamically override theme properties using a ConfigProvider. The downside of this feature is that only some properties are actually customizable, which means this solution may not be sufficient for our use case. We would also need to upgrade Ant Design to the latest version and that would require adjustments to our application to make sure the styles are still working. The advantage of this approach is that we won’t expand our layer of custom components, preventing work that may be discarded when upgrading to Ant Design 5. Also, it would be a catalyst to finally get visual testing integrated into our development environment. Migrating to the latest version would also help prepare the code base for the next major version upgrade.

Complete the layer of custom components

Another possibility is complete our layer of custom components by adding mappings to Ant Design components that are currently being accessed directly. This has the disadvantage of expanding our custom layer even more which may change after upgrading to Ant Design 5. The advantages are that we don’t need to upgrade Ant Design, avoiding all the migration errors, and the level of possible customization.

Wait for Ant Design version 5

We can also wait for the next major version and limit the scope of what we can customize. During the POC it was hard to determine if the areas that were not affected by theming changes are the result of code that is not using the theme correctly or if it’s related to Ant Design styles.

Make use of global styles to override specific components

We could also use global styles to fill in the blanks and override specific components while keeping the Ant Design version. This could be a temporary patch until we wait for version 5.

Conclusion

I think the decision of which approach to follow should be made after we fix incorrect code and check the remaining non-themeable parts of the application. Then we can fully determine which approach is faster, given that the new Ant Design version will probably change the solution.

Having visual regression testing enabled for this work would be essential to mitigate potential regressions. We should make it a pre-requirement for many of the efforts described here.

Apply our theme to ECharts plugins

ECharts supports theme customization by providing and registering a theme object or using a predefined theme and explicitly setting style options.

Here’s a preview of the ECharts theme structure obtained by downloading the dark theme:

var colorPalette = ['#dd6b66','#759aa0','#e69d87','#8dc1a9', ...];
var theme = {
   color: colorPalette,
   backgroundColor: '#333',
   tooltip: {...},
   legend: {...},
   textStyle: {...},
   title: {...},
   toolbox: {...},
   dataZoom: {...},
   timeline: {...},
   timeAxis: {...},
   logAxis: {...},
   valueAxis: {...},
   categoryAxis: {...},
   line: {...},
   graph: {...},
   gauge: {...},
   candlestick: {...}
};

In our case, we recommend using the default theme and mapping our theme configurations to each category defined by ECharts. Currently, each plugin is passing these ECharts configurations via an options object that is calculated by the transformProps and loadTransformProps functions. Right now, each plugin is creating its own version of the options object. To allow customization of properties that are applied to all plugins we can initialize the Echart component with a base options object and complement its properties with the ones coming from the plugins. We also need to remove base configurations from the plugins unless we’re specifically overriding the behavior. Here’s an example:

// src/components/Echart.tsx
function Echart({ echartOptions, ... }: EchartsProps) {
   ...
   const theme = useTheme();
   const chartRef = useRef();
   const option = {
      backgroundColor: theme.colors.background.default,
      textStyle: {
         ...theme.typography.presets.default,
      },
      ...echartOptions,
   };
   chartRef.current.setOption(option, true);
}

// src/Radar/transformProps.ts
export default function transformProps(theme, ...) {
   const echartOptions: EChartsCoreOption = {
      ...,
      radar: {
         splitLine: {
            lineStyle: {
               color: [theme.colors.grayscale.base],
            },
         },
      },
   };
   return {echartOptions, ...};
}

As you can see from the examples, we need to make the Superset theme available for both functions.

Apply our theme to non-ECharts plugins

To apply our theme to plugins that are not using the ECharts library, we’ll need to evaluate each plugin individually and determine what’s the best approach, whether is to manually apply CSS classes to the relevant parts of the plugin (e.g. Pivot Table) or use the plugins’ API to change theming properties (e.g. Word Cloud).

Create a dark theme to test the application

Creating a dark theme will be a good test for Superset themeability. Using a theme with opposite colors enables us to validate if all UI elements are being affected by theming changes. This is especially useful to check text, borders, shadows, and background colors with different contrasts. We could also make the dark theme available as one of the default themes after the work is completed.

Create a wiki page on how to customize Superset

Organizing the theme structure helps a lot but there’s nothing better than a proper theming guide for Superset users. We’re suggesting a wiki page with all the information about theme categories, which parts of the application are affected by each category, instructions on how to manage themes, etc.

Future work

In the future, we could extend the work to allow administrators to manage multiple themes via a specific module in Superset. We could have features similar to ECharts theme builder or Material UI theme creator for editing theme configurations. We could also allow administrators to export/import themes to promote theme sharing.

New or Changed Public Interfaces

• We’re suggesting changes to the theme object definition, which will impact multiple React components
• Migrating the server-side rendered pages may result in endpoint changes. This will probably be defined in its own SIP
• The plugins will receive access to the theme object as an additional property

New dependencies

Depending on the output of our research on how to apply our theme to Ant Design, we may have to upgrade its version.

Migration Plan and Compatibility

No database migrations or updates to stored URLs are necessary.

Rejected Alternatives

• Keep theming variables in CSS or LESS files. We want to leverage Emotion and its dynamic features.
• Remove the atomic typography categories and keep only the presets. We don’t want to remove the developer’s flexibility as new categories may emerge.
• Keep server-side rendered pages and make some sort of theming sync. After analyzing the complexities and limitations of the solution, we decided to migrate them to the client-side since this is also a requirement for the single-page application (SPA) effort.

Special Thanks

Special thanks to @kasiazjc @geido @rusackas @mihir174 @jess-dillard @villebro and all the other reviewers for helping with the work.

[ktmud]

I understand big migration is a huge risk but since we are talking about theming here, I’ll still encourage you all to try out the theming approach in Chakra UI (which is inspired by Theme UI, btw), maybe even just for inspiration of how to apply themes to AntD. It’ll blow your mind.

[michael-s-molina]

This is really cool! Thanks for sharing this @cdmikechen.

I totally understand the need to support advanced customizations. Think of this SIP as the first chapter of a book about Theming 😉. Before adding advanced features, we need to fix our foundation, but eventually, we’ll get there! 🚀