[RFC] Charts & EUI: Forking vs Merging-in vs Folding-in

Charts & EUI: Forking vs Merging-in vs Folding-in

Elastic UI Framework (EUI) is a collection of React UI components used throughout the interfaces of Elastic’s web products and Elastic Charts is a set of chart components used within EUI and Elastic’s web interfaces. Both EUI and Elastic Charts were built specifically for use in Kibana and primarily by employees of Elastic, and have historically been maintained in the open via Github. They have clear contribution guidelines, component development guides, component design docs, and style guides. Both EUI and Elastic Charts are the core UI and Visualization frameworks that comprise the bread-and-butter of the core OpenSearch Dashboards experience.

The UI framework used to be a separate package, named kibana-ui-framework, until it was merged into Kibana in 2016 for faster iterations, version locking, and unified styling. Late in 2017, a new React-based look-and-feel was created which resided in a repository separate from Kibana, named EUI, and some parts of Kibana started using it. The code for ui-framework eventually got moved to an internal package of its own within the Kibana source code and until just recently, Kibana contained components from both of these sources to support plugins that hadn’t gotten updated to use EUI.

The purpose of this document is to review the recommendation for resolving the recently announced licensing constraints on Elastic Charts and EUI, both dependencies used in OpenSearch Dashboards. OpenSearch Dashboards uses older versions of these libraries and to be able to develop new features and fix bugs, the project would need to create a copy of the last open-sourced version of them; this can be done in a couple of ways which are discussed below.

Principles

1. The solution should be easy to maintain
2. Should have minimal disruptions to existing workflows
3. Should not increase developer friction
4. Should not limit future expandability

The Recommendation

None of the main options have any strong contraindications for adoption and none violate the principles unpleasantly.

• Merging-in is unnatural and ugly, with a potential for complications if code leaks into other parts of Dashboards which makes this the worst option.
• Forking, while being the universally accepted way, forces the project to maintain yet another package which can get complicated if the community’s needs are not supported. Also, without adding features unnecessary for Dashboards but required to be competent, the fork could get the Razzies.
• Folding-in as an internal package is a common pattern in Dashboards. It saves the project from having to maintain one more package and yet, it allows for completely separating the code from that of Dashboards’.

Considering the pros and cons, the recommendation for the Charts and UI is Folding-in; this is a two-way door and reversible with minimal effort.

Options

Elastic Charts and Elastic UI Framework (EUI) are hosted in individual repositories and get pulled into OpenSearch Dashboards as NPM dependencies during installation. Currently in Kibana, updates to these packages get built and released individually to NPM and a version bump tracks the iterations; Kibana is then updated to reference the new version. Dashboards also has a built-in package manager, @osd/pm, which is inspired by Lerna but works with Yarn, link-ed dependencies, and workspaces to have fine-grained control. This package manager was built for several reasons, mainly to get away from wastefulness of webpack and towards individual in-production distributable bundles:

Long-term we want to get rid of Webpack from production (basically, it’s causing a lot of problems, using a lot of memory and adding a lot of complexity). Ideally we want each plugin to build its own separate production bundles for both server and UI. To get there all Kibana plugins (including x-pack) need to be able to build their production bundles separately from Kibana, which means they need to be able to depend on code from Kibana without import-ing random files directly from the Kibana source code.

— The Kibana project management tool

The options considered in this document are to either fork and publish which is the natural NPM/Yarn way, to pull in the code and merge it in with the existing code, or to use Dashboards’ package manager which was built for this specific purpose.

Folding-in

This would involve pulling in the code for Charts and EUI, and merging them into the packages folder of OpenSearch Dashboards where they will be consumed by the built-in package manager. Kibana uses its built-in package manager for over 50 packages and OpenSearch Dashboards which inherited the built-in package manager from Kibana, uses it for over 20 packages. Hence, this will not be alien to Dashboards developers and contributors, and should not disrupt their workflows.

Plugin developers would link (NPM, Yarn) the project’s UI components to their packages during development which involves a one-time execution of a command or two; if included unambiguously in the getting-started documentation, this would eliminate any real friction. Additionally, this will unify how developers use any of Dashboards’ packages since all will be distributed as @osd/pm packages; Charts and EUI will no longer be among the exceptions that gets imported differently.

The code for Charts and EUI, residing in the packages folder, will be entirely self-contained, complete with their own tests and package.json which would make it as easy, if not more, to maintain when compared to having the code in their own repositories - since no extra building and publishing is needed. There is also precedent for folding packages into Kibana, datemath, which could help reduce the long term maintenance of Dashboards by reducing the effort cycles needed to publish it. Also, being self-contained, folding-in will not impose any limits on any future expanding.

On the other hand, even though the code will be open-sourced, by being nested inside Dashboards, Charts and EUI will be prohibitively expensive for the community to consume as a dependency and as such, the opportunity to have them independently contribute to open-source is lost. However, since an absence of the Charts and EUI libraries will not leave behind a gap, it is fair to assume they won’t be missed. The effort for Dashboards to consume the code from its new location would be minimal, perhaps limited to a string find-and-replace.

Forking

Forking the packages under OpenSearch Project would allow the project to maintain the status quo: plugin developers and contributors can continue to use their existing workflows to develop and test their plugins and so will OpenSearch Dashboards; this causes no disruption. While this allows for plugin development to occur independent of the underlying Dashboards, these plugins cannot function fully outside of OpenSearch Dashboards and need to be loaded for complete execution and testing. Additionally, plugin developers who choose to distribute certain versions of these packages in their code will potentially face conflicts during installation or runtime, or force extra payloads on to their users - via loading duplicate libraries for similar functions and degrading performance.

Forking and publishing these packages would improve the project’s standing with yet another contribution to the open-source community. However, adding features requested by the community that don’t specifically serve the needs of OpenSearch Dashboards would cost the project in effort while not addressing them would stain its reputation.

That said, until recently, Elastic Charts and Elastic UI Framework (EUI) had a warning on their repositories against others using these packages for anything and as such, no non-related package consumes Charts and only a couple consume EUI. Considering that these components are specifically made to serve Kibana’s at times unique needs, developers have better generic options for their UI components like Material-UI, React Bootstrap, and Ant Design.

To implement this option, the project would need to fork two repositories, change some names, reset the versions, push the changes, and publish to NPM. Afterwards, with every change that is pushed, a version bump and publishing to NPM will be required. However, consuming a published-to-NPM package is natural for JS developers - by simply including it as a dependency in their own package configuration; this adds no developer friction. Maintaining separate repositories, this method doesn’t impose any limits on future expandibility.

Merging-in

To avoid having to maintain separate build and release pipelines, the project could simply pull in the code for Chart and EUI, and merge them into OpenSearch Dashboards. The code will mix with that of Dashboards and this entanglement will make it a little more complex for the project to maintain. With minimal exposure, using exports, plugin developers can link the project’s UI components to their packages during development (NPM, Yarn). This involves a one-time execution of a command or two which if included in the getting-started instructions, would add very little disruption to the workflows developers employ; beyond that, it should be transparent to plugin developers.

This option differs from Folding-in in that the code, tests, and packaging logic for Charts are mixed in with those of Dashboards’; this could potentially complicate any future changes the project may want to make to how plugins share assets with Dashboards. Merging in just increases the number of source files in the Dashboards’ code which is only ugly; if the code is thrown into a specific folder of its own, it wouldn’t be any more difficult to maintain or expand than having it in a separate repository or package; in fact, one could argue that there would be nothing to maintain anymore since there would be no extra build and publish steps needed.

On the flip side, even though the code will be open-sourced, by being nested inside Dashboards, it takes away an opportunity for the project to contribute to the open-source community but the space already has strong contributors and a gap will not be felt. This would require a non-zero initial effort to implement; string replacement to the references would be cheap but testing, building, and production-packaging will need customization.

Others

There are a few other options that are simply unreasonable or complicated, and they are briefly mentioned here for the sake of completeness.

• Build new charting and UI libraries using opensource options or from scratch: The project would get to design, build, and document a proper library that matches its needs, employing the latest techniques and latest advanves. This would be a huge undertaking that would have a very long lead time. The project would need to build an entire community around these new libraries which will force drastic changes in plugins’ UI. this is the “boiling the ocean” approach.
• Use a combination of forking and folding in: If the project starts by forking the repositories and then uses git subtrees to nest them in as Dashboards’ @osd/pm packages, it would get to contribute to the open-source community and also enjoy all the benefits of folding in. However, using git-subtrees will contribute to developer friction by slightly complicating their keeping up with remote changes or contributing back; Atlassian has documented some drawback of git subtrees here.

[galangel]

just my 2 cents. I’m ok with having it as yet another package, but it will be a huge one keep in mind. so I’ll give it my vote =)

there are pros and cons to everything but, I still hold the opinion that we should fork it. and maintaining yet another repo isn’t so bad.

In my vision, the OpenSearch components library should be separate, and this style guide should be one of the faces of the project, I would like devs that have no relation to OpenSearch to use this library, and they could offer unbiased perspectives and improvements. if osd has unique needs it’s something else that we should address.

By limiting its usage only to OSD in favor of control we are limiting the possibility of growth. I can’t predict the future but I hope this will evolve into its own thing.

[JacobBrandt]

@galangel Honestly, I could get behind a fork of EUI as well. It just seemed like that was a hard no from the team.

EUI can hold its ground against the other React based UI component libraries. The point that there is a warning about using EUI outside of Elastic doesn’t mean much. FWIW, Elastic still talks about the Kibana plugin API as an experimental, use at your own risk, feature. I think the main concern from the team about forking was who would manage/own this fork. The OpenSearch team is right in that it takes extra work (not just development) for them to do it. However, it doesn’t have to be an OpenSearch repository 😉 . When MapboxGL changed its license Elastic didn’t create an Elastic fork. The open-source community created MapLibre and now Kibana depends on MapLibre.