[RFC] TechDocs Addons Framework

Status: Open for comments

Need

TechDocs is a centralized platform for publishing, viewing, and discovering technical documentation across an entire organization. It’s widely adopted in and outside of Spotify and already solves the core author/view/explore use-cases well.

Having had TechDocs in place for quite some time at Spotify, TechDocs has evolved significantly in order to tackle higher-order needs internally: it doesn’t just show documentation, it shows documentation quality. It doesn’t just enable documentation authoring, it encourages a documentation culture. The manner in which these higher-order concerns are handled is via augmentations (now called “addons”) to the core TechDocs experience.

A common pattern with existing Spotify-developed addons is to collect metadata that is only (or is most conveniently) available at build-time, store it in an aggregated flat file, and make it available to the TechDocs frontend. As an example, the “Last Updated” addon retrieves the latest build timestamp and displays it on the header of the site to indicate how up-to-date the content is.

While Spotify has authored a set of such addons, it is by no means the definitive set; we anticipate other adopters would likely choose to address their own higher-order concerns in their own unique ways (whether using Spotify-authored addons, community-contributed addons, their own custom addons, or a combination thereof).

The exact framework that enables this at Spotify can’t be directly externalized as it relies on and makes assumptions about technology unique to Spotify. However, in introducing a framework appropriate for the open source TechDocs community, we aim to:

• Enable read-time augmentation of the TechDocs experience at the GLOBAL-, SITE-, and PAGE-level.
• Enable build-time metadata capture and augmentation that can optionally be leveraged by addon frontends
• Ensure the framework is ubiquitous (meaning, it can be used in both out-of-the-box and recommended setups, including when using TechDocs CLI as a preview/editing tool)
• Provide a delightful developer experience (e.g. a robust, documented, testable API surface area)

In addition to enabling addons, TechDocs already contains a variety of built-in features as of today provided by MkDocs:

• In-context search
• Side menu of the left
• Generating static HTML from Markdown
• Various rich content: LaTeX, Graphviz, PlantUML, Mermaid (provided by mkdocs-techdocs-core)
• Table of contents on right
• Navigation for previous and next pages on bottom

Implementing a general addon structure would allow segmenting each core functionality into its own addon to make all features configurable, testable and separated by concern.

Proposal

Vertical layers of addons (backend endpoints, and CI/CD hooks)

With this RFC, addons will first only concern frontend behavior. In the future, we should allow addons to simply expose backend endpoints ideally by leveraging existing Backstage backend APIs to avoid introducing new concepts. These endpoints can be used for dynamic metadata and content, such as daily generated reports.

For CI/CD steps, as to enrich TechDocs metadata with available data, addons should then also be able to provide this at build time. The data should be flat, and when processing at build time we should write data to techdocs_metadata.json with the addon name as key. Then we provide TechDocs addons with metadata e.g. via context.

When packaging a new addon, we should be able to provide all frontend, backend and CI/CD functionality in one package.

Static site generator agnostic

Currently, TechDocs is highly dependent on MkDocs due to inspecting, manipulating and extending content by relying on MkDocs specific selectors in transformers, reader page and addons. In addition to that, there is a potential hard dependency to material-ui v.4 using StylesProvider.

We started to work on separating MkDocs as one individual content provider (PR). This emphasizes that instead of TechDocs adapting to MkDocs internals, we aim to provide a common interface for TechDocs content layout, and hook MkDocs generated content into it by creating addons for each core feature.

Draft implementation

Internally at Spotify, we have implemented a working draft framework to port existing additions in TechDocs at Spotify to open source. At this point, it is a frontend-only API with several locations in the element tree (so called “Addon Locations”, see below) into the TechDocs reader page at runtime. It leverages Backstage’s plugin and extension API and is configured by passing addons with props to the app routes. Furthermore, not all addons need to visible on all pages on all sites. Therefore it is possible to configure “Addon Scopes” for each addon, and where applicable for each site and page.

Additionally, there are testing utilities to emulate a fully rendered TechDocs reader page with shadow root which will be provided in a separate package.

TechDocs Addon Scopes

• GLOBAL: Render specified addon for all TechDocs pages
• SITE: Allow TechDocs sites to opt-in into displaying an addon on all pages of its site
• PAGE: Allow TechDocs pages to opt-in into rendering addon instances within the content of its page

TechDocs Addon Locations

Locations are separated into “permanent” and “virtual” groups:

Permanent Locations

• HEADER: Filling up the header from the right, addons can be added on the same line as the title
• SUBHEADER: Between header and above all content, tooling addons can be inserted for convenience
• PRIMARY_SIDEBAR: Left of the content, above of the navigation
• SECONDARY_SIDEBAR: Right of the content, above the table of contents

Virtual Locations

• CONTENT: Allow mutating all content within the shadow root by transforming DOM nodes. These addons should return null on render.
• COMPONENT: An instance of the addon is rendered for every HTML node with the same tag name as the addon name in the Markdown content. If no reference is made, no instance will be rendered. Works like regular React components, just being accessible from Markdown.

API by example

In practice, this is how an addon goes all the way through to be rendered:

// Foo.tsx
export const Foo = ({ bar }) => bar;

// plugins.ts
export const FooAddon = techdocsPlugin.provide(
  createTechDocsAddon({
    name: 'FooAddon',
    location: TechDocsAddonLocations.HEADER,
    component: Foo,
  }),
);

// routes.tsx
<Route path="/:namespace/:kind/:name/*" element={<TechDocsReaderPage />}>
  <TechDocsAddons>
    <FooAddon bar="baz" />
  <TechDocsAddons>
</Route>

Configuration matrix

Depending on each addon location and addon scope, there is a need to configure each combination. The table below categorizes the intended options given to both app integrators and documentation creators.

	GLOBAL scope	SITE scope	PAGE scope
Virtual location COMPONENT	(not applicable)5	(not applicable)5	Register in scope2 + Use addon tag in Markdown content4
Virtual location CONTENT	Register in routes1	Register in scope2 + register in mkdocs.yml3	Register in scope2 + Use addon tag in Markdown content4
Permanent locations (all others)	Register in routes1	Register in scope2 + register in mkdocs.yml3	Register in scope2 + Use addon tag in Markdown content4

1) Register in routes

<TechDocsAddons>
  <FooAddon bar="baz" />
<TechDocsAddons>

2) Register in scope

<TechDocsAddons>
  <FooAddon scoped bar="baz" />
<TechDocsAddons>

3) Register in mkdocs.yml

plugins:
  - techdocs-addons:
      addons:
        - FooAddon:
            bar: baz

4) Use addon tag in Markdown content

# This is a heading

This is a paragraph

<FooAddon bar="baz" />

5) Not applicable

Since addons in the virtual location COMPONENT can be placed anywhere in the content, there is no pre-defined point identical neither for all pages in one site nor across all sites. Therefore adding them in GLOBAL and SITE scopes has no effect and is discouraged.

Testing

The testing utilities in its own package can be used for the example addon above like this:

// Foo.test.tsx
describe('Foo', () => {
  it('renders without exploding', async () => {
    await buildAddonsInTechDocs([<FooAddon bar="baz" />])
      .renderWithEffects();
    expect(screen.queryByText('baz')).toBeInTheDocument();
  });
});

Presets

By splitting up the core features into individual addons and by aggregating addons improving similar experiences and use cases, we can bundle multiple addons into “presets”, e.g. techdocs-addons-preset-core or techdocs-addons-preset-feedback (see below).

Example selection of addons contributed by Spotify

To illustrate how addons can be used in the real world, the following is a selection of 3 (+1 hypothetical) addons to improve the feedback loop on content to help keeping documentation up-to-date, relevant and correct.

// plugins.ts

// appears in the header and shows when the doc site was last updated
export const LastUpdatedAddon = techdocsPlugin.provide(
  createTechDocsAddon({
    name: 'LastUpdatedAddon',
    location: TechDocsAddonLocations.HEADER,
    component: LastUpdated,
  }),
);

// provides a count of the total number of issues in the repository (whether a docs with code repository or docs only repository)
export const IssueCounterAddon = techdocsPlugin.provide(
  createTechDocsAddon({
    name: 'IssueCounterAddon',
    location: TechDocsAddonLocations.SECONDARY_SIDEBAR,
    component: IssueCounter,
  }),
);

// enables you to highlight text and add a GitHub Issue
export const GiveFeedbackAddon = techdocsPlugin.provide(
  createTechDocsAddon({
    name: 'GiveFeedbackAddon',
    location: TechDocsAddonLocations.CONTENT,
    component: GiveFeedback,
  }),
);

// hypothetical addon: allow voting for readers by providing a question and multiple answers
/*
export const InlinePollAddon = techdocsPlugin.provide(
  createTechDocsAddon({
    name: 'InlinePollAddon',
    location: TechDocsAddonLocations.COMPONENT,
    component: InlinePoll,
  }),
);
*/

// routes.tsx
<Route path="/:namespace/:kind/:name/*" element={<TechDocsReaderPage />}>
  <TechDocsAddons>
    <LastUpdatedAddon />
    <IssueCounterAddon />
    <GiveFeedbackAddon templateBuilder={bugTemplate} />
    {/* <InlinePoll scoped /> */}
  <TechDocsAddons>
</Route>

Alternatives

Using MkDocs plugins

At build time both the content and layout of a TechDocs page can be altered. It is possible to insert addons as static counterparts to all locations except HEADER (and arguably SUBHEADER as well). The counterparts are required to be HTML only as any additional HTML tags would otherwise be filtered out by the DOM sanitizer.

This would work for static content that only needs to be updated when the site rebuilds.

Keep addons frontend-only

By expanding the concept of addons to include backend and CI/CD entry points, we improve the developer experience for addon developers for the cost of increased complexity and amount of public interfaces.

In most cases it is possible to retrieve information at runtime. This would incur additional performance overhead every time a documentation page is rendered, but it would be possible to retrieve the desired data.

Not using MkDocs / static site generators at all

Instead of taking the output generated by MkDocs and adding features on top of it, we could implement the desired product from the ground up, without limitations from the design of current MkDocs pages. This would allow full flexibility at the very high cost of re-implementing TechDocs as a whole along with its external contributions.

Risks

Generally speaking, all parts of TechDocs that interact with the DOM assume certain MkDocs internals. That is, node structure, class names and data attributes. Therefore, by aiming to make TechDocs fully independent of the implementation details of MkDocs, nearly all parts have to be changed, ranging from smaller adjustment to large architectural redesigns (e.g. this addon framework).

One large conflict that arises is that currently we are moving within two ecosystems: Python with MkDocs and TypeScript with React and Material UI. By moving towards a JavaScript based addon approach, we actively distance ourselves from the Python ecosystem, as we intend to mainly focus on the TypeScript ecosystem. The main drawback would be needing to port existing MkDocs plugins or existing core functionalities, that in some cases might be hard to re-implement.

[iamEAP]

Thanks for following along, everyone! We’ve made significant progress on the Addon framework initially proposed here. …Documentation for which is about to be merged and deployed. Although things are still under alpha exports and the next release line, it’s now possible to experiment with Addons in your Backstage instance!

Our plan is to properly introduce the Addon framework to the community with the v1.2 release of Backstage (coming up in just a few weeks).

I’ve opened follow-up issues that cover portions of this RFC that have not been implemented (see linked issues just above this comment). The @backstage/techdocs-core team will likely shift focus to @backstage/plugin-search for the next month or two, so we won’t be pushing the framework forward ourselves during that time. But if you’re interested in helping out, please join in on conversations in those issues! We always dedicate time to engage with / encourage community contributions.

I’ll follow-up in that mermaid issue separately with some pointers on how it could be implemented.

[tragiclifestories]

Just want to offer another upvote for this. We have customised our mkdocs stack somewhat but it always feels kind of hacky; I’m never sure what techdocs ‘assumes’ about the output of the mkdocs build and if I’m breaking it or something. In that resepect - despite how much work it would likely be - I’m somewhat sympathetic to the ‘rewrite the whole thing in TS’ approach, which would also have the benefit of removing a whole language and packages from my 5gb docker image 😱 But anything that allows us to make these little customisations within the app will be great.

As an example, the “Last Updated” addon retrieves the latest build timestamp and displays it on the header of the site to indicate how up-to-date the content is.

People are always asking for this! In particular for runbook type documents the ‘freshness’ of the content can be very important, and this is something that you get in confluence/google docs/etc but not in techdocs unfortunately.

People are also always asking for MermaidJS support but seems like that’s already being discusses over in #4123 ❤️