TechDocs: Deprecate legacy `git clone` and dir preparer in favor of URL Reader

What?

When TechDocs is set to generate docs on the Backstage server, it has to “prepare” the source files before it can generate the site. In the early days, TechDocs implemented a simple git clone way of downloading the repository. If your backstage.io/techdocs-ref annotation is not prefixed with url:, but has github: or gitlab:, TechDocs will use the legacy git clone preparer. The dir: is a syntactic sugar for the git preparer.

Later on, URL Reader received the feature of readTree by which it can read files and trees by downloading an archive of the repository.

Why deprecate?

1. URL Reader (downloading archive) is much faster than Common git preparer (a git clone). TechDocs preparer step only needs markdown files and not the git history of the repository, which could increase the size 10x for a repository as large as Backstage mono-repo (Prepare times reduces from 1 minute to 5 seconds)
2. URL Reader is used across Backstage and not just TechDocs. Thus, it is much actively maintained and receives frequent support. Common Git Preparers are maintained by TechDocs and only used in it.
3. URL Reader supports more source code host providers (GitHub/GHE/GitLab(and hosted)/BitBucket(and hosted)/Azure DevOps (and on-premises)) and their intricacies like different API versions than Common Git Preparers in TechDocs. Using URL Reader adds support for GHE, Bitbucket, etc. It’s quite likely that URL Reader will continue to be stable and exhaustive with more providers.
4. dir: preparer is complicated to maintain alongside url: preparer (and is only used and maintained by TechDocs). I feel that the syntactic sugar is not worth the effort of maintaining it for just one plugin. url: is standard across backstage.

How to migrate?

The software templates in this mono-repo already uses the url: format in backstage.io/techdocs-ref annotation in the Entity’s catalog-info.yaml.

If you have created your own Software Templates, which use github:, gitlab:, dir:, etc. for backstage.io/techdocs-ref annotation. - update them with url: format. You also need to update existing catalog-info.yaml that have been created and use TechDocs annotation. Here is a simple HOW TO guide to help you https://backstage.io/docs/features/techdocs/how-to-guides#how-to-use-url-reader-in-techdocs-prepare-step

Note that dir: was sometimes used for local development when working with TechDocs. Now, we can use https://github.com/backstage/techdocs-cli for preview when writing docs locally and to debug any build errors. However, an alternative will be to keep dir preparer but not maintain it for production use-case.

Plan

Throw warning in backend logs when non-URL preparer is used, with the migration link. Release on Day X.

In the release after X + 30 days, drop support for git and dir preparers in TechDocs.

Open for comments.

[OrkoHunter]

In the value of backstage.io/techdocs-ref annotation (url:TARGET) the TARGET can only be an absolute URL, but it makes sense to support the use case of url:./ or other relative URLs.

Quoting @freben for implementation suggestion

In some places we support that and use the ScmIntegrationProviders stuff to help resolve into an absolute URL I think the logic in that case would be, start with a base of the annotation backstage.io/source-location if it exists and otherwise backstage.io/managed-by-location, and then resolve the backstage.io/techdocs-ref on top of that

[dhenneke]

I provided an implementation of the relative url reference in https://github.com/backstage/backstage/pull/6263 to discuss the issue regarding relative urls and the destiny of the dir: location there.