[PROPOSAL] Introduce an OpenSearch-Dashboards SDK

Is your feature request related to a problem? Please describe.

Around 2015, Kibana started offering a plugin system, but no public plugin APIs. By 2020, before the OpenSearch fork, it has launched the Kibana Development Platform that significantly improved plugin development, including foundational APIs, and runtime isolation. Plugins can register HTTP endpoints and UI applications, query and create back-end data, and provide generic services to other plugins.

Post-fork, the major problems in extensibility that remain unsolved are as follows:

1. Developers of plugins are forced to release new versions of plugins for each patch version. This is terrible developer experience and makes it very difficult for users of OpenSearch Dashboards to upgrade as they must wait for all the plugins they use to become available, which may be never.
2. Developing plugins requires checking out and building a specific version of OpenSearch Dashboards, which is fairly sizable and rather … specific. For all the runtime isolation and independence of plugins, developers are forced to bring in a very specific version of the kitchen sink.

Describe the solution you’d like

tl;dr An OpenSearch Dashboards SDK, similar to OpenSearch: https://github.com/opensearch-project/opensearch-sdk-java.

Since OpenSearch adopted semver at fork it may seem like we can now relax the exact version check (problem 1), and assume that a plugin developed during 2.3.0 will work with the next release of OpenSearch Dashboards, 2.4.0. This is because per semver, as plugins reach deep into @osd/core/server we can safely assume that those interfaces don’t change between 2.3.0 and 2.4.0. Or do they? Well, maintaining semver in all OpenSearch Dashboards interfaces seems quite difficult, and relies on maintainers to be extra careful about changing anything. TODO: @kavilla a specific example pls?

An SDK solves this by providing a logical and physical separation.

An SDK provides a logical separation. The SDK will only contain the set of APIs that need to follow semver, significantly reducing the surface of APIs that OpenSearch Dashboards needs to worry about since plugins only take a dependency on the SDK. Backwards incompatible changes can be immediately caught by testing every API in an already shipped SDK against all released versions of OpenSearch Dashboards (including the next version). Thus, we guarantee backwards and forwards compatibility. With a semver-stable SDK plugins should be able to confidently declare that they work with OpenSearch Dashboards >= 2.3.0 or ~> 2.5. The SDK could provide support for integration testing against those versions. Finally, an SDK can begin selecting common functionality that all plugins may need, such as storing credentials or saving objects, and be strongly opinionated about what constitutes a semver-compatible extension point for OpenSearch Dashboards.

An SDK provides a physical separation. The SDK would be much smaller than Dashboards. Thus, to develop a plugin on top of an SDK, no need to check out and build the kitchen sink, much less a specific version of the kitchen sink. The SDK can be published to npm, and follow its own semver, and document only the public interfaces it contains. It can also implement wrappers that translate for multiple major versions of OpenSearch Dashboards, extending compatibility much further, enabling developers to write plugins once for several versions of OpenSearch. Testing your custom plugin should be done against a released, downloaded, and stable version of OpenSearch Dashboards.

There are downsides to an SDK: it’s a standalone additional project. It will have to forward/copy code/interfaces. What else?

Describe alternatives you’ve considered

1. Relax version checks.
2. Mark extension points clearly and continue relying on humans to follow semver.

Additional context

I propose we call plugins that use the SDK extensions to make it easy to identify those by a set of new capabilities (broad version compatibility), and match how OpenSearch is thinking about extensibility.

The remaining questions are:

1. Plugins can also be extended. There are plugins that are part of OpenSearch dashboards that would need to also publish interfaces into the SDK. And there are plugins that will be using the SDK that will want to publish their own public interfaces and SDK. As a plugin developer I want to be able to say: my plugin needs OpenSearch Dashboards ~> 2.5 and alerting ~> 7.3 (alerting 7.3+ works with any version of OpenSearch Dashboards 2.x).
2. The security model can/should/will be moved behind the SDK and offered by the SDK. Where/how/what?

Some related issues:

• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2584
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2544
• https://github.com/elastic/kibana/issues/135781
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2507
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2145
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2065
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/2036
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1866
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/1724
• https://github.com/opensearch-project/OpenSearch-Dashboards/issues/287
• https://github.com/opensearch-project/opensearch-build/issues/606

[seraphjiang]

for extensibility, I’m looking forward to see SDK extension experience like https://vscode.dev/

[kavilla]

Developers of plugins are forced to release new versions of plugins for each patch version. This is terrible developer experience and makes it very difficult for users of OpenSearch Dashboards to upgrade as they must wait for all the plugins they use to become available, which may be never.

They actually don’t have to if they set a reserve word in the opensearch_dashboards.json. For the key opensearchDashboardsVersion they can set to opensearchDashboards. And it will work, but devs need to trust that there plugin works.