[RFC] TechdocsX (MDX support in techdocs)

Status: Open for comments Given my lack of context, please point out anything that I might miss or have gotten wrong.

Need

In its current form, techdocs allows markdown and it relies on plugins to add extra functionality. MDX has been growing in popularity in the past few years and would allow enriching our docs experience in a developer friendly way.

The rationale is as follows:

1. The augmented hero’s journey Stories are more personal and engaging when interactive. Following the dawn of the interactive news stories , perhaps the time has come for ~docs~ stories as well.

2. React is ubiquitous MDX is simply React. All developers with React knowledge would be able to contribute with extra functionality. Additionally, markdown plugins add ad-hoc syntax that is largely unknown to developers and hard to teach through documentation. With MDX, the syntax would be the same for all extensions. Additionally, documentation for the available components can be embedded in the docs, themselves, with… MDX.

3. Writing MDX components is easier than integrating MD plugins To integrate plugins today, one has to extend the core functionality of techdocs. With MDX the available components can be externalized, both by providing a standard library and allowing ad-hoc contributions for 3rd party Backstage installations (see proposed solution below).

4. Innovation speed and wow effect The point of this proposal is not to list all things we could do with MDX (although a modest list is provided below), but point to the fact that once this is enabled, we could expect increased and faster contributions from both internal and external users. A great step in becoming a leader in developer experience.

Proposal

To integrate MDX with our current offering successfully I envision the following requirements (please comment if there are important requirements missing):

• Be as backwards compatible with the current implementation as possible.
• To be able to render MDX remotely and in a safe way. Ideally, we shouldn’t move any javascript down the wire and do not rely on any evaluation of code.
• Ensure the components follow design guidelines.
• Document the feature extensively.

The proposed solution to the above is as follows:

1. Standard library Create a standard library for the available components (say @backstage/mdx-components). These are maintained by us and follow our design guideline (requirement 3).

2. Backward compatibility When creating a project, developers can still use only MD. Since MDX is mainly a superset of MD, these would be consumed almost the same way they are today (requirement 1). A bit of extra research would be needed to map out all the breaking changes, but they are known to exist mainly when rendering HTML inside MD and this is not a particularly hard problem to solve.

3. Incremental adoption Developers can opt in to use MDX in their repos. The way they would do that would be by installing our component library and registering the components implicitly (as in the components property of the MDX provider) in their MDX renderer (that is, no import syntax). Since we would only allow for these components, no ad-hocs components written in the MDX files would work. Locally, since the components are available docs would simply work and render the components correctly.

4. Security The MDX content in these files would move down the wire in their text representation. That is, not as JSON or anything fancy, literally as, e.g. <PieChart data={[{mdx: 70, md: 30}]} /> (requirement 2).

5. Rendering The rendering side (aka Backstage) would then interpret the text representation and render the components from the same library. This is already done in the wild, for instance with the next-mdx-remote package provided by Hashicorp. Although the name mentions nextjs, the library can actually be used elsewhere and includes functionality for server side rendering, client side “hydration”, context providers (so that components can have shared state and additional functionality) and more.

6. Documentation The standard lib can be documented in MDX itself and display its components, usage, types and props, how to contribute, etc (requirement 4).

Possibilities

An image is worth a thousand words, so here are some examples of things that MDX could enable:

• Components with state.

  • Variables in docs, as in {your-service-name}. With MDX one could ask the user to enter the service name once and update all the instances of the variables in the docs. Or these variables could even be taken from the user context (say {your-username}) or be remotely fetched. One interesting example of such a thing are the dropdowns in asdf docs and others.
    
  • Allow users to write personal comments, highlight or pin content to read later like in pocket.

• Simple ad-hoc forms that today would require contribution to Backstage. One could build polls, feedback forms, quizzes that could be integrated in instructional material like in NextJS docs, etc.

• Embedding of different file formats, beyond graphviz.

• Graphs & charts and dataviz that could use local or remote data like in nevo docs.

• Live coding for instructional docs or even more ambitious sketchbooks like in observablehq.

• Enhanced functionality for code blocks, like smart copy/paste for diffs, link to remote, auto GH search… Code blocks and highlight could be handled by MDX instead of rendered previously, allowing for more languages faster and other things.

Risks, downsides, other thoughts

• Versioning of the standard libraries. That is, if a doc uses an incompatible version of a component it could break the renderer. A problem to solve.

• MDX can’t be immediately previewed at github, github enterprise, bitbucket and most other systems with some MD support since it relies on extra compilation steps. The MD parts of it are still visible 'tho.

• Backward compatibility could prove to be a bit harder than anticipated, given the current dependency on mkdocs and the interaction with plain HTML in MD.

• Indexing of MDX content.

• Alienating users. If a teams with both people who can write React and people who can relies on MDX too much it may become a barrier to contribute from those who don’t, although I imagine teams mainly falling into 2 buckets: the ones who simply use MD and the ones who leverage MDX and most members can maintain it.

• There’s a need for teams to publish their own component libraries docs. Given this would involve ad-hoc components, this is not detailed here, but a similar solution apart from techdocs could be considered.

• It is conceivable that for many simpler use cases, MDX docs would be a simpler solution than writing backstage plugins themselves. I can foresee developers writing simple components as part of their docs instead of writing and integrating a full blown plugin.

• A future Backstage desktop app could interact with the standard components, and given proper permissions, perform automation tasks, like running a command automatically in the terminal (waiting for its conclusion and checking the tutorial step), opening an app and performing a task, etc.

[andrewthauer]

I wouldn’t see any issues if both syntaxes are supported for this (MDX & old school code fences). I’ve used remark in the past, and it’s quite powerful with a very extensive eco-eystem. So being able to tap into that would be ideal.

It does feel a little like this proposal is overlapping with what gatsby and next.js provide though. Prior to adopting Mkdocs, I looked at using Docz which provided a lot of similar features built on top of gatsby. However, I found it required much more setup, whereas mkdocs was simple to run and build. Lliterally a mkdocs.yml file and your off to the races. I’m hoping whatever solution is chosen the same is true. I’d love to be able to point to a directory of markdown files stored somewhere and have it work as a site.

On a side note, we are currently using the Mkdocs awesome-pages plugin which allows for navigation customization with .pages sidecar files. We’ve been able to avoid having a central nav tree and dynamically generate the mkdocs.yml on the fly so it’s not required to exist in a repo.

[Rugvip]

Wondering if a version where MDX for a specific set of known components is useful. The TechDocs frontend plugin would be configured with both built-in and custom components, and those can then be used in MDX. Gives us a way to provide more powerful doc features, and a way for integrators and organizations a way to power up their docs as well, but without the headaches of dynamically loading in things and the security issues that come with it.

The components you want to make available in MDX would probably be provided with some strict prop-types definitions, likely a subset that only allows primitive values.

Possibly also combine with a bit of data collection in the backend so provide data on what components and props are in use and by what docs, allowing an organization to detect broken docs and plan migrations.