[Design] Drag & Drop Visualization CX

Background

Today, visualization and exploration in OpenSearch Dashboard is harder than it needs to be. Users need to preselect both the chart type and index pattern they will use to create a visualization. Additionally, users may need to navigate between multiple screens (e.g. discovery → visualize → discover → visualize, etc.) to find the right feature, chart, index to include in their dashboard. Not only is these more work then it should be, it prevents the serendipitous discovery that easy exploration allows. This document outlines the high level design recommendation for a visualization experience that makes exploring data and creating visualizations much easier.

Tenets

• Minimal learning curve
• Easy of use
• No disruption to existing user workflow
• Get feedback early
• Progress not perfection

Out of Scope

• This design only covers the initial version of the tool and should not be seen as final
• This is only a high level design and doesn’t include component-level design

Recommended Design

The recommendation is to create the Drag and Drop experience as a first party plugin within the Opensearch Dashboards repository (i.e. explain to layman). It will act as an additional way to create visualizations alongside the exiting tools within the current visualizations plugin. The tool will be incremental to the visualization tools available to the user in OpenSearch Dashboards today. Due to the large nature of the design, I will divide it into 4 major sections:

• The user flow
• Tool features
• Architecture
• Aggregation logic

While we considered many options for each of our choices, we ultimately recommended this approach so that we could have minimal disruption to existing users workflow while also providing a seamless experience to the users who wish to create visualizations easily. It also lets us iterate and evolve over time.

The user flow

There are 3 main ways the user can reach the new tool.

1. Through the “New Visualization” option within the Visualize page.
2. Via the edit option for a visualization created by this tool in the dashboard
3. From the saved object viewer

For each of these flows the user is navigated to a full page view of the drag and drop tool that lets the user create or edit the visualization. After creating the visualization, the user can either choose to save the visualization or add the visualization directly to the preferred dashboard (existing or new).

Alternatives

1. Upgrade the existing visualize plugin to support the drag and drop: This approach is to update the exiting visualize plugin to be more easy to use and support some of the features this design proposes.
   1. Pros:
      1. No need to maintain two plugins
   2. Cons:
      1. Limits the scope of the features we can introduce. Some features such as the need to select the the visualization type before hand or the need to know the type of aggregation first go against the features introduced by this design. Changing it to align with the design breaks the experience for users that rely on the current experience.

Tool features

This tool features:

1. Top Nav: To search for data within an index pattern. This is identical to the top nav element within the Discover plugin and allows the user to search and filter the index pattern data for available fields and its data. It also allows the user to save the visualization as a saved object or save and add it directly to a dashboard.
2. Data Panel: A panel for accessing the Index pattern and its available fields. The data panel will feature an index pattern selector, a search bar to filter all the fields available within the index pattern for the given Top Nav filters, and the index pattern fields broadly subdivided (e.g. categorical fields, numeric fields and meta fields). The main features of this panel are:
   1. Index pattern selector: To select the index pattern
   2. Field filter: Filter the list of available fields. Helpful if the list of available fields is very large.
   3. Field summary: Each field, when hovered over or clicked, will show us a summary of the data in the field, similar to the discovery experience.
   4. Add to canvas: There will also be a button within each field to add the index pattern to the canvas.
3. Chart type selector: To allow the user to switch chart types on the fly, we will have a chart type selector. The initial implementation will prioritize bar (vertical and horizontal + stacked), line, area, and pie as well as metric and data table.
4. Visualization Config Panel: The panel is where the fields to visualize are added. It also has options to configure the type of aggregation for a field and serves as drop targets for the draggable fields.
5. Visualization Style Editor: A panel to edit the overall styles of the visualization such as the title, axis labels, legend, etc.
6. Visualization Canvas: The canvas serves as an area to see the final visualization. It also doubles as a drop target for the different draggable fields if a suitable aggregation is possible.
7. Drag and drop: All fields available in the Data Pane can be dragged and dropped into either the Visualization Canvas or the Visualization config panel. Doing so triggers an appropriate aggregation call described in the Aggregation section.

Architecture

Workflow

The app uses the fields selected or dropped into the chart context by the user to determine the best aggregation logic (outlined in a later section) to generate the appropriate DSL Query(s) and fetch the data to render the visualization.

Saved Object & Embeddable

The tool will introduce a new saved object type and embeddable to render its visualizations. This means that existing visualization will not be compatible with the new tool and separate factory. This is done to prevent breaking existing visualizations while allowing new visualizations to be created easily without much overhead.

Saved object structure

const wizardApp: SavedObjectsType = {
  name: WIZARD_SAVED_OBJECT,
  hidden: false,
  // ....
  migrations: {},
  mappings: {
    properties: {
      title: {
        type: 'text',
      },
      description: {
        type: 'text',
      },
      // Contains the state of the visualizaion
      state: {
        type: 'text',
        index: false,
      },
    },
  },
};

The one big drawback to this approach is not having the ability to edit existing visualizations using the new experience. Since the existing experience has a lot of visualization types that it already supports and a lot of customizations that we will not support for the initial launch and in future, reusing the existing schema will be rather tedious.

If the ability to edit existing visualizations is important, we can optionally provide an api/interface to selectively convert or migrate exiting and compatible visualization saved objects to the new saved object type. This will duplicate the existing visualization to the new format so that if there are any issues with the migration, the existing visualization remains unaffected. This approach also allows us to selectively enable migration for the visualizations that can be safely supported.

Alternatives: Reuse existing visualization schemas: This approach lets us keep the existing schema and all new visualizations created using the new experience uses the existing visualization saved object type and embeddable. Supported visualization types will open the new experience when edited while the old experience is used for unsupported visualizations. This allows existing visualizations to use the new experience easily and allows wider adoption with no migration required, but not all options supported by the existing experience will be supported in the new experience. This may break workflow for current users. Additionally, this would break existing visualizations when edited and some existing visualizations may never be supported by the new experience (e.g. Vega). Lastly, inheriting a schema that is not optimized for the current experience may limit future expansion.

Charting Library

The app will use the EUI Charts library to create the default supported visualizations. This to to maintain visual parity with existing visualization while changing how they are generated. In future we expect all visualization to be created using this app.

While the default visualization types use EUI, there is no restrictions on what charting library can be used since each visualization type will be responsible for rendering itself and defining the fields and config types it supports it need to render correctly. In future this method can be used by other plugins to register their own visualization to exist alongside the default visualization types.

Drag and Drop

For the initial release there will only be 2 primary drop targets:

1. Visualization Config Panel
2. Visualization Canvas

The field cards can be dragged from the Data panel into either drop target and the page will automatically provide context to the user based on where the user is dragging the field.

Based on the dropped region an appropriate aggregation strategy will be used to fetch the data and render a visualization is possible. When a particular field cannot be dropped into a particular region, the drop cue does not display.

Aggregation Logic

One of the key problems we will need to address with the tool is how different interactions render a corresponding visualization that is both useful and makes sense to the user. Each existing visualization tool out there today does this in their own way and have their own strengths and drawbacks. To simplify this problem for our initial release and to get better user feedback on the approach we need to take here, my recommended is to allow the user to only be able to drag and drop a field into the canvas for the first field they wish to visualize. The Visualization generated will be based on a simple lookup below.

Field Type	Visualization Canvas
Numeric	median
Date	count of records (histogram)
String and others	count for top values

The Visualization Config panel will update with the appropriate fields and aggregation functions and the Visualization canvas will no longer act as a drop target. All subsequent fields need to be added directly to the Visualization Config Panel which assumes an appropriate aggregation function for the Chart axis and field type (similar to the existing visualize tool today). The User will also still be able to configure the visualization by Dragging and dropping fields directly onto the Visualization Config Panel. This approach makes the initial discovery of the data easy while keeping the reason for why a particular visualization was rendered intuitive.

Alternatives

1. The smart alternative: Similar to Tableau and Lens, we can allow the user to drop in any valid combination of fields into any valid drop target to generate an appropriate visualization. The biggest challenge here is to breakdown the problem such that it makes intuitive sense to the user why a particular visualization was rendered given the fields they dropped in. Given the number of drop targets, field types and aggregation functions Opensearch supports, the number of permutations quickly explode. Hence this approach is not recommended for the first iteration
2. The super simple approach: In this approach, we do not allow the user to drop anything into the canvas directly, but instead only allow them the pick the field they want to visualize directly in the Visualization config editor. This removes any ambiguity about what field and aggregation is used to render the visualization. It is not ideal since the user will now have to rely on the field summary in the data panel to explore the data, but it is still better than the experience today since the rest of the tool still makes intuitive sense.

The recommended solution is a meets both these approaches in the middle.

Future scope

1. Replace the most existing visualization tools and make this the default
2. Allow integration for other query languages
3. Migrate existing visualizations to the new tool
4. Ability to drag and drop any field into the canvas to create visualizations
5. Realtime smart suggestions for visualizations

Open Questions

1. Should we prioritize converting visualizations to work between the current visualizations experience and the proposed experience?
   1. Is the ability edit existing visualizations using the new experience more important?
   2. Is the ability to edit a visualization created using the proposed experience with the current experience important?
   3. Are they both important?
2. Visualization types to prioritize:
   1. Bar
   2. Line
   3. Metric
   4. Data
   5. Area
   6. Pi
3. How can we provide a preview or beta.
4. Is there anything in the future scope that we should add to the initial release
5. Any core feature we have missed for the initial iteration
6. Should this be a part of Opensearch Dashboards, the Dashboards-visualizations plugin, or be a separate plugin

[jgough]

In the list of Available Fields to select from it would be extremely useful to replicate the Popular section as used on the Discover tab:

This is an extremely useful feature for both novice and advanced users as it helps them easily locate the fields they require most often very quickly and easily.

[JacobBrandt]

Im only seeing index pattern as an option for the data panel. I’ve mentioned several times before that saved searches should be there as well. Especially since allowing the user to choose a saved search would have no impact to the generic design I had hoped was going to be built. Removing this ability would make this the only visualization that can’t use saved searches just like Kibana did with Lens and Maps and seems needlessly limiting. Is the user going to be able to choose from a saved search as well?

The app will use the EUI Charts library to create the default supported visualizations. This to to maintain visual parity with existing visualization while changing how they are generated. In future we expect all visualization to be created using this app.

Wait, maintain visual parity with existing visualization? The majority of OpenSearch Dashboard visualizations DO NOT use EUI charts. When this was forked from Kibana, EUI Charts was barely used. I believe, if I remember right, only Discover and some of TSVB used EUI Charts. If you mean to say visual parity with latest Kibana, okay I can buy that. But is this visual parity requirement an Amazon need or a community need?

There’s a lot to consider when bringing in another Elastic built repository now behind non open source licenses besides where the library will be hosted (this repo or separate). For instance, what are the number of bugs/regressions that are being inherited by creating EUI Charts from the version that can be forked? What’s the plan to fix all of these regressions and bugs that have been inherited? Has the team considered the missing features in Lens that are missing because of its use of EUI Charts, which in turn limits this tools potential in the future? I’m curious, what alternative charting libraries were discussed with the community and team?