[SIP-34] Proposal to establish a new design direction, system, and process for Superset

[SIP-34] Proposal to establish a new design direction, system, and process for Superset

Motivation

In August, 2019, a project was kicked off to re-imagine the design of Superset, and create a “North Star” for the interface, flows, and visual aesthetic of Superset. The primary goal of the project—and this SIP—is to create a new visual design framework for Superset’s user interface.

Since its inception, Superset has rapidly grown in scope, size, audience, and complexity thanks to the work of this community. This growth has led to an accumulation of “design debt” in forms including problematic user flows, inconsistencies in interface elements, or general complexities that may be simplified/clarified by revisiting the design of Superset holistically.

The intent of this undertaking was to smooth out the user experience of Superset, targeting all of the following:

• Supporting most use cases and functionality currently in use by known Superset users in the design process, while reducing dead-ends and friction in these flows. Features and flows removed or deprecated are listed under “Removals and deprecations” below
• Focusing on the “objects” of superset (namely Datasets, Queries, Charts, and Dashboards) and how those are structured/accessed throughout the flows of Explore and SQL Lab
• Smoothing out and simplifying the general experience, making Superset easier to use for new users, and faster for more experienced ones
• Creating a more consistent design and experience throughout the product, with a well defined style guide, and coherent design system

Participants in this project included:

• Preset, Inc. - a startup founded by @mistercrunch to offer Superset as a hosted service, acting as primary sponsor of this design effort
• Cartel - a digital product design firm focusing on research, strategy, and UX design for the next generation of enterprise software companies
• Various members of this community, including Airbnb, Lyft, and others

We hope that this new design direction will carry Superset forward for the foreseeable future, and create a unified design system upon which to build future features. Community feedback is welcomed, and when this SIP is voted in, further efforts will be undertaken to break down the design into smaller units of work.

Process used in this effort:

Efforts began with the announcement of a call for volunteers in a design interest group, via the Superset dev mailing list, and on the Superset Slack workspace. The search for members of this small group targeted individuals who were able to commit 2+ hours per week, have a history of involvement/merit with the project, have a sense of design, and understand Superset users and use cases.

A “Superset Design Group” email was set up, as well as a private channel on the Superset Slack workspace, to maintain an open line of communication throughout the duration of the project

A kickoff meeting of the resulting group was held on September 4th, 2019. This initiated the creation and iteration of core user stories. This was quickly followed by an effort to assess the Analytics / BI marketplace, resulting in a competitive analysis report (this research is proprietary, and cannot be shared publicly).

Weekly design sync meetings took place each Thursday, allowing Cartel to get feedback on their latest design proposals from the team/community, and gather insight/direction on “big question” issues about the product’s direction. After each session, prototypes were circulated, and feedback gathered via the slack and email channels. Each 1-week design sprint was comprised of ongoing work on larger concepts (e.g. the relationship between SQL Lab and Explore, or controls used to build data visualizations) as well as smaller, more specific tasks.

Summary of key findings

A product audit of Superset, user personas, and general product recommendations are available in an Experience Brief.

Special thanks

This process involved significant time and involvement from the following members of the Superset community for their participation:

• Grace Guo (Airbnb / @graceguo-supercat)
• Vyl Chang (Airbnb)
• John Bodley (Airbnb / @john-bodley)
• Sylvia Tomiyama (Airbnb)
• Eli Brumbaugh (Airbnb / @elibrumbaugh)
• Michelle Thomas (Airbnb / @michellethomas)
• Maxime Beauchemin (Preset / @mistercrunch)
• Rob DiCiuccio (Preset / @robdiciuccio)
• Aaron Suddjian (Preset / @suddjian)
• Tai Dupree (Preset / @nytai)
• Beto Dealmeida (Lyft / @betodealmeida)
• Micah Delfino (Cartel)
• Jef Bekes (Cartel)

Proposed Changes

Notable changes to design and functionality

Visual Design

• All UI fonts are changed to Inter UI for increased legibility, and fixed-width numeral options for tabular data. Code fonts have been changed to Fira Code. Both of these fonts are open source (license information below).

• New default color palettes for dataviz. These have been checked for accessibility issues. Old palettes will be supported for backward compatibility.

• A full style guide is available in the prototype, with specs for a variety of UI element interaction states, layout spacings (all based on an 8 pixel grid)

  Style Guide (see abridged InVision)

Simplified global navigation.

• Links to Home (see next item), Dashboards, Charts, Explore, SQL Lab, and Data

  Current	New

Dashboards

• New list/card view of all dashboards with ability to favorite dashboards, filter them by owner/status/dataset, search by name, sort by various metrics, or perform bulk actions.

• New card view displays cards with rendered previews of the dashboard

• Individual dashboards include new tools for each chart indicating data recency and filters applied

• Edit a chart’s display options directly from the dashboard

• New tab UI with dropdown overflow menu when there are too many tabs to display

• New UI modal to define/edit global filters, with bulk action support

  Current	New

New “Home” page

• Offers the ability to view Dashboards, Charts, and Queries, sorted by popularity, recency, favorites, and ownership.

• Activity steam, showing who did what in a Superset instance

  Current	New

Explore

• Visualization type selector, with viz plugins broken out into categories (e.g. timeseries, distribution, composition, etc)

• Left panel for dataset selection and resulting display of fields

• Updated query panel, with visualization controls utilizing new “pills” containing field metadata and controls (e.g. aggregation, sorting)

• New Save/Save As controls, highlighting potential implications from overwriting an existing chart

• Dataset preview area, including data profiling in the column headers (i.e. histograms, null value count, statistical rollups, etc)

• New Chart Options drawer in the top right, focused on visual aspects of the resulting chart

• New UI to edit a dataset’s source, fields, metrics, and more

• History timeline with icons denoting query “run” and “save” events

• New Time Picker input modal, with quick selection of various time ranges and granularities

  Current	New

SQL Lab

• Left panel has tabs for Data (a picker for databases/tables/fields) and Saved (a selector for saved Queries)

• Data preview table with headers containing data profiling information (histograms, composition, etc)

• Chart area to build quick/simple data viz to preview query results and/or add viz to a dashboard. Viz options here are limited in scope/complexity. If your needs expand beyond these simplified viz options, there’s an option to move over to Explore.

• Explore button, allowing users to take their SQL queries into Explore for more advanced viz options with expanded controls

• History panel with icons and data-rich tooltips indicating when queries were run or saved, and the details thereof

• Full Query history with timeline, query preview, and option to group by session

  Current	New

Data section

• Contains sortable/searchable/filterable card or list views of Datasets (i.e. “physical” static/tablular datasets or “virtual” dynamic query results), Databases, Saved Queries, and Query History. Each of those tabs displays various metadata (e.g. access statistics) and actions (edit, delete, etc) for the resulting items

• Viewing a Physical or Virtual Database displays its Fields, Metrics, and Usage tabs displaying various statistical and historical/analytic metadata details

  Current	New (Datasets)	New (Fields)

Charts

• New list/card view of all charts with data source display, ability to favorite items, sorting (e.g. by time last updated), and filtering by ownership and status.

  Current	New

Global Search

• Improved search and discovery of dashboards, queries, dataset titles, and possibly more

• Results can be filtered by type

• Default (prior to typing) list of recent or commonly accessed items

  Current	New (default, no input)	New (search results)
  N/A

Design proposals, in the form of InVision prototypes, are encapsulated in the deliverables available here:

• Select screens (key features and concepts)
• Unabridged screens (contains some details yet to be entirely fleshed out)

Removals and deprecations

• Tabs in SQL Lab have been removed (See related SIP).

Maintenance and improvement of design assets

For maintaining the new design work (and assets stemming from it), the current proposal is to maintain the creation of a Superset team on Abstract. The latest Sketch files will be made public and open source, and managed via Abstract’s version control system. PMC members from the Superset community may reach out to the PMC mailing list to submit designers or design teams to be added to the Abstract team to contribute to the design. The latest master version of the files will be maintained in the public InVision prototype, maintained by Preset (though interested parties may request comment or write access, granted on a case-by-case basis).

User testing and validation plan

Design and feature proposals submitted herein were conducted using the Superset Design Group as a focus group, making sure that functional and visual changes to the UI address the issues and use case criteria of those involved in the discussion. It is our intent, and belief, that these designs push the usability of Superset forward greatly. We anticipate that individual features and UI elements will require further input and discussion, as this “north star” for design begins to play out. Our hope is that this round of design will provide a foundation to test further iterations/variations, and those results will bear out in additional SIPs and PRs on a case-by-case basis.

Updates to Superset documentation

Documentation and Screenshots for Superset exists in a number of places, most notably on the Apache website and on the Github repo. A best effort should be made to update these resources with updated screenshots and/or instructions as relevant as PRs are written to move the project toward this new UI. If any changes are encountered that require large scale changes to documentation, those changed should be discussed in an additional SIP.

New or Changed Public Interfaces

TBD — Impact will be minimized on existing interfaces, but implementation of workflows and features may require changes to how APIs and metadata are structured and/or accessed.

New dependencies

The scope of third party dependencies is not entirely known at this time, but will be defined and assessed in future PRs as the work is broken down into smaller tasks. An effort will be made to make sure all licenses are compatible with Apache License v2.0.

Some known licenses thus far include:

• Inter UI (font) — SIL OpenFont license (needs to be labeled in ASF releases)
• Fira Code (font) — SIL OpenFont license (needs to be labeled in ASF releases)

Migration Plan and Compatibility

Due to the broad scope of design and functional changes, several core sections of the codebase — both frontend and backend — will need to be modified in functionality and/or refactored. This will likely lead to a significant number of required database migrations. While breaking changes in backward compatibility will be made as minimal as possible, they may occur in subsequent PRs as this work is further scoped and ticketed, and should be tracked with semantic versioning in future releases.

For frontend implementation, it should be noted that an older version of React-Bootstrap is used in existing Superset UI components. This older version is built upon Bootstrap 3, which is built with LESS (as is much of Superset, allowing shared styles/variables). Upgrading components to a newer React-Bootstrap (and thus Bootstrap 4) would require migrating the bulk of the CSS codebase over to SASS, which is not ideal. The implementation of this SIP should iterate toward deprecating the React-Bootstrap dependency, in favor of bespoke components with custom LESS styles built around this new design system.

All changes, in the design and implementation of new components, should be “atomic” in nature, steering toward a more unified design/component system, targeting reusability and consistency.

Rejected Alternatives

No alternatives at this time. We’ve considered previous design-related SIPs in this process, and have reached this proposal with input from several active PMC members. We’re open to feedback, however, so please feel free to leave constructive feedback below. This work is to be considered foundational, and will undoubtedly continue to evolve with the ongoing support of the Superset community.

[richlysakowski]

I am very concerned that the proposed major re-designs proposed (and approved) appear to add significant complexity for SuperSet administrators and end users. Also, a big concern about a push to “custom components”, rather than more standardized open source components provided with the open source kit now. Statements by @rusackas “whittling away react-bootstrap using custom components” and by @ktmud “if we are to build all UI components by ourselves” suggest that this is what is underway. @rusackas “The implementation of this SIP should iterate toward deprecating the React-Bootstrap dependency, in favor of bespoke components with custom LESS styles built around this new design system.” The comment by @gbrian “merge SQLAB and Chart Explore” seems to decomponentize the design; it does not retain separation of GUI concerns for the user.

Deprecating major technologies like React-Bootstrap in the name of “simplification” for developers will introduce yet another reactive framework, and delay the user community getting a version 1.0 product. It adds a lot of complexity for end users who just want to stand up something quickly that scales in performance and functioality, and is highly usable.

This complete redesign completely torpedoes a lot of standard functionality provided “out of the box” that is required to almost effortlessly stand up a running system with minimal effort. We want a completely working system that includes all servers, subsystems, data connectors, and a decent looking GUI that requires NO customization to get running, and provides tool to customize from a working system.

This product has been “incubating” for several years already. The proposed redesign will require Apache “re-incubation” for another year or two. If it ain’t broke, don’t fix it.

What is really going on here? It appears like commercial professional software consultancies wanting to add complexity to a product to guarantee job security for staff, or perhaps guaranteeing that SuperSet v1.0 never finishes incubating, and always requires experts to set up. This is reminiscent of the ways that Oracle, IBM, and Microsoft wage in IEEE, ASTM, and ISO fight standards wars by putting technical staff on IEEE, ASTM, and ISO standards committees to guarantee that a standard specification is never implemented. The likely culprits in this advanced analytics and business intelligence open source war for software are Tableau, Tibco, Microsoft, and other BI advanced analytics vendors who don’t want more open source competition. They would love to stall Superset more or less permanently.

Users DO NOT WANT a complex expert-driven procedure to stand up a Superset system or tailor it.

We need a SIMPLER process to stand up a Superset instance, NOT MORE COMPLEX. We need a simple one-step installation to get up and running and configuring “sensible default” dashboards. Something as simple as “docker pull” followed by “docker run”, point your browser here, and open your first visualization.

Remember to target end users (analysts and dashboard assemblers, not programmers) in 95% of all work that you do, because that’s who going to be using it the most.

Finish incubating and shipping v1.x of SuperSet before a doing total redesign.

[john-bodley]

Thanks @rusackas et al. for putting this great synopsis together.