API Extractor Support for Docusaurus V2

Feature Suggestion

To prepare for migrating to Docusaurus V2 some changes will need to be made to the API extractor so that the generated references are supported. The way front-matter is parsed has changed slightly, and more notably documents are now parsed as MDX. Some scenarios that result in parser exceptions include:

1. Front matter with symbols like title: @backstage/app-defaults

Can't process doc metadata for doc at path "/backstage/docs/reference/types.md" in version "current"
Loading of version failed for version "current"
YAMLException: end of the stream or a document separator is expected at line 3, column 8:
    title: @backstage/app-defaults

1. Use of empty comments, <!-- -->

SyntaxError: /backstage/docs/reference/types.observable.md: Unexpected token (19:66)
  17 | };
  18 | `}</code></pre>
> 19 | <b>References:</b> [Observable](/docs/reference/types.observable)<!-- -->, [Observer](/docs/reference/types.observer)<!-- -->, [Subscription](/docs/reference/types.subscription)
     |                                                                   ^
  20 |     </MDXLayout>
  21 |   )
  22 | };

1. Missing closing tags for some HTML elements

SyntaxError: /backstage/docs/reference/plugin-catalog-react.useownedentities.md: Expected corresponding JSX closing tag for <Entity>. (35:34)
  33 | <b>Returns:</b>
  34 | <p>{`{ loading: boolean; ownedEntities: `}<a parentName="p" {...{"href":"/docs/reference/catalog-client.cataloglistresponse"}}>{`CatalogListResponse`}</a>{`<`}<a parentName="p" {...{"href":"/docs/reference/catalog-model.entity"}}>{`Entity`}</a>{`>`}{` `}{`|`}{` undefined; }`}</p>
> 35 | <p>{`CatalogListResponse`}<Entity></p>
     |                                   ^
  36 |     </MDXLayout>
  37 |   )
  38 | };

Possible Implementation

1. When writing front-matter, we can quote the value.

context.writer.writeLine(`${name}: '${value}'`); 

https://github.com/backstage/backstage/blob/master/scripts/api-extractor.ts#L511

1. We can remove the empty comments, possibly adding a .replace(/<!-- -->/g, '') to the getEscapedText method.

2. The getEscapedText method is already replacing pointy-brackets with their encoded counterparts, but some of these are still making it to the resulting document. Some more troubleshooting is required to determine why this is the case.

Context

This is one-step in preparing to migrate the Microsite to Docusaurus V2 – the overarching issue can be found here: #8329.

[cmpadden]

FYI, there is an issue that discusses the compatibility of the output of api-extractor and the new MDX parser of Docusaurus.

https://github.com/facebook/docusaurus/issues/2708#issuecomment-623932672

Which states:

This is not a bug but a limitation/specification of MDX/JSX. GitHub uses Markdown which allows HTML, but Docusaurus 2 uses MDX which requires that your tags are all closed. Few ways:

• Write a postprocessing step which closes the violating tags
• File an issue on microsoft api-extractor to get them to close the tags

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.