[Proposal] Consolidate chart-rendering to Vega-Lite

Is your feature request related to a problem? Please describe.

Today, there are at five different charting libraries that are use to render different visualization types in OpenSearch Dashboards (OSD). This makes it difficult for developers to provide a consistent user experience across visualization types, because any vertical feature that targets all visualizations must be implemented separately in each.

Essentially we want to

1. Provide OSD users with a clean, unified charting experience that looks polished and consistent.
2. Provide OSD developers with unified interfaces to develop, enhance, and interact with charts

Describe the solution you’d like

1. All new OSD features and improvements will target Vega-Lite as the rendering layer and Vega-Lite as the declarative grammar to describe chart attributes and interactions
2. OSD will provide high-level APIs and React components will make it easy for plugin authors to use/add/extend charting features without interacting with Vega-Lite directly
3. Design system styling and interaction patterns will be built-in by default, so that developers will don’t need to re-implement them
4. OSD will formalize charting contracts and capabilities for plugins that require alternative rendering libraries or approaches

Describe alternatives you’ve considered

1. Instead of choosing a single preferred visualization library, just focus on wrapper APIs, components, and contracts
2. Introduce Plotly.js as a new dependency and choose as the charting library for all visualization rendering
3. Introduce Apache Echarts as a new dependency and choose as the charting library for all visualization rendering

Additional context

Why Vega-Lite?

Vega-Lite is built on years of academic research to create a concise, declarative JSON specification for interactive visualizations that enables users to rapidly create interactive visualizations. Even complex combinations of charts, layers, and interactions can be constructed by merging or updating a single simple object structure. It also makes it easy to write API wrappers, because they only need to generate JSON output (or partial output) following the Vega-Lite specification.

OSD already has a dependency on Vega-Lite, and it’s one we can’t easily remove even if we chose a different charting library (like Plotly.js). The fundamental value of the Vega visualization feature is that it allows end user’s to build and create totally new chart types by writing Vega/Vega-Lite specifications directly so there’s no sensible way to migrate that rendering to another library with different syntax.

The existing Vega visualization feature also provides a foundation for a low-code visualization creation flywheel. Users can use the built-in Vega-Lite editor to prototype new chart types not currently supported in OSD, and share these with the community, and it will be easy to “graduate” these chart types to Vis Builder or other easy-to-use charting tools. In the other direction, we can provide the Vega editor as an “advanced” editor to tweak or change the behavior of officially defined visualizations. This approach provides greater flexibility and power than existing systems which only allow custom JSON configurations without an interactive environment.

Vega-Lite visualizations have already been connected into all the important plumbing of OSD - they can be rendered in dashboards, have expressions implementations, and even some styling to make them similar to the built-in visualization types, so much of the groundwork has already been laid.

Some developers may initially find the Vega-Lite grammar to be a bit different from other JavaScript charting libraries or APIs, but that’s part of its expressive power. The best way to get a sense of it is to look at examples.

To see Vega-Lite’s approach to defining interactivity, see this example of a threshold based annotation/overlay with binding. (Note the use of params, which is Vega-Lite’s system for setting and using variables within the declarative data structure.) See the gallery for other examples.

To directly compare it to more traditional JavaScript charting libraries like Plotly.js, here’s a multi-series line chart with a unified tooltip and guide implemented in both:

1. Vega-Lite
2. Plot.ly

How we decided

We did a quick survey of 20+ JavaScript charting library APIs, adoption, and usage to narrow-down our decision to four contenders: Vega-Lite, Vega, Plotly.js, and Apache Echarts, each of which met most of our criteria, summarized in the chart below (you may need to scroll right to see the entire table).

-	Vega-Lite	Vega	Plotly.js	Apache Echarts
URL	https://vega.github.io/vega-lite/	https://vega.github.io/vega/	https://plotly.com/javascript/	echarts.apache.org
Description	Vega-Lite is a high-level visualization grammar. It provides a concise JSON syntax for supporting rapid generation of visualizations to support analysis. Vega-Lite support interactive multi-view graphics. Specifications can be compiled to Vega.	Vega is a visualization grammar, a declarative language for creating, saving, and sharing interactive visualization designs. With Vega, you can describe the visual appearance and interactive behavior of a visualization in a JSON format, and generate web-based views using Canvas or SVG.	Built on top of d3.js and stack.gl, Plotly.js is a high-level, declarative charting library. plotly.js ships with over 40 chart types, including 3D charts, statistical graphs, and SVG maps.	Apache ECharts is an open-sourced JavaScript visualization tool, which can run fluently on PC and mobile devices. It is compatible with most modern Web Browsers, e.g., IE9/10/11, Chrome, Firefox, Safari and so on. ECharts depends on ZRender, a graphic rendering engine, to create intuitive, interactive, and highly-customizable charts.
Repo	https://github.com/vega/vega-lite	https://github.com/vega/vega	https://github.com/plotly/plotly.js	https://github.com/apache/echarts
Actively maintained	yes	yes	yes	yes
Latest version (date)	5.22.1 (Mar 2022)	5.4.0 (Jul 2022)	2.14.0 (Aug 2022)	5.3.3 (Jun 2022)
GH stars	3.9K	10K	15K	52K
dowloads/week	300K	300K	200K	500K
Version in OSD	5.17.3	4.16.8	1.57.1, 2.2.0	N/A
License	BSD 3-Clause	BSD 3-Clause	MIT	Apache 2.0
Adher to SemVer
Documentation	Docs	Docs	Reference	Handbook
Roadmap	Roadmap	Roadmap	None	None
Security Posture
Min add’l bundle weight	0 (already in bundle; 91kB)	0 (already in bundle 172.4 kB)	1.1 MB minified + gzipped	321.8kB minified + gzipped
Delarative API	yes	yes	yes	yes
facets/subplotting	yes	yes	yes	yes
layering/compositing
bindable interactivity	yes	yes	yes	yes
Rendering	SVG, canvas, HTML5	SVG, canvas, HTML5	SVG, WebGL	SVG, canvas, WebGL, VML
SSR	yes	yes	yes	yes
responsive	yes	yes	yes	kind of
a11y features	aria labels	aria labels	no	automatic labels
i18n	yes	yes	yes	yes
react library	no	no	yes	no

[anirudha]

The Observability plugin needs the following requirements reviewed as we migrate to the abstraction interface.

need to list all plugin requirements for visualization & overlays need to formalize existing visualization requirements that are used today so plugins can easily migrate need a process to request new features for visualization and overlays need to review the data layer ( that follow associative vectors or a data frame structure )

[anirudha]

Priorities

need to start working on an Abstraction interface , and the following requirements are met.

need to list all plugin requirements for visualization & overlays need to formalize existing visualization requirements that are used today so plugins can easily migrate need a process to request new features for visualization and overlays need to review the data layer ( that follow associative vectors or a data frame structure )

Decisions / Questions

1. What frameworks do the anywhere projects use to implement layering on Dashboards.
   1. Anywhere projects should not use a charting library but should use an “Abstract Charting Interface”.
2. What does the Abstract Charting interface use as its impl. library ?
   1. There will always be a need for a multiple charting library, but for base charting types we need to ensure we have standard features supported. This is today supported by Vega-Lite, Plotly and Echarts
3. There is no-substantial technical difference in the charting implementations, the minor difference can be fixed in each option as we become contributors and start working on visualizations.
4. How do we decide between libraries,
   1. Features we need ?
      1. Custom Charting support “VEGA” visualization [ important ]
      2. Multiple data sources
      3. Long-term community support / Issues , PRs
      4. Governance model, Ability to become maintainers
      5. Performance
5. How much weight do the legacy implementations carry in our charting library decision

in Summary,

• based on the above, VEGA is an important visualization type and has existing support/usage in dashboards.

• Echarts has a higher community adoption, but does not easily enable custom visualizations and user created visualizations similar to VEGA plugin. E-charts is an apache foundation project that has a more promising long-term impact on the project and governance.