TechDocs URL Reader method won't work with Azure DevOps Server 2020

Expected Behavior

TechDocs should still generate using the URL Reader method with Azure DevOps Server 2020

Current Behavior

When opening a Docs site that is configured to use the URL Reader format using Azure DevOps Server 2020 you get a 404 error page suggesting that perhaps you are missing an index.md but if you look at the browser console there is a 404 error loading an index.html from an API route. Opening that route gets you to this error:

NotFoundError: Failed to read tree from https://ados-servername/organization/project/_git/repository, 404 Not Found at AzureUrlReader.readTree (webpack-internal:///…/backend-common/src/reading/AzureUrlReader.ts:106:15) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async UrlReaderPredicateMux.readTree (webpack-internal:///…/backend-common/src/reading/UrlReaderPredicateMux.ts:59:16) at async getDocFilesFromRepository (webpack-internal:///…/techdocs-common/src/helpers.ts:243:20) at async DocsBuilder.build (webpack-internal:///…/…/plugins/techdocs-backend/src/DocsBuilder/builder.ts:81:25) at async eval (webpack-internal:///…/…/plugins/techdocs-backend/src/service/router.ts:151:13)

Possible Solution

From walking through the code what we’ve found is that the URL is not being parsed correctly for Azure DevOps Server. This URL comes from the getAzureDownloadUrl function here:

https://github.com/backstage/backstage/blob/d941f6187b6490878cfea67a46ce573130659765/packages/integration/src/azure/core.ts#L108

What we are seeing is that both the organization and project variables contain the same values like this: organization/project/_git. You could potentially fix it with something simple like this:

if (url.includes('dev.azure.com'))
{
   return `${protocol}://${resource}/${organization}/${project}/_apis/git/repositories/${repoName}/items?recursionLevel=full&download=true&api-version=6.0${scopePath}`;
}
  
return `${protocol}://${resource}/${organization.replace('/_git','')}/_apis/git/repositories/${repoName}/items?recursionLevel=full&download=true&api-version=6.0${scopePath}`;

Steps to Reproduce

These steps assume that you are using Azure DevOps Server 2020

1. Create a valid catalog-info.yaml file and add it to your repo
2. In your catalog-info file add a backstage.io/techdocs-ref annotation using the URL Reader format for it’s value: url:https://ados-servername/organization/project/_git/repository
3. Make sure you have a \docs folder with at least an index.md
4. Add a mkdocs.yml file in the root of you repo with this as the body of the file:

site_name: 'example-docs'
nav:
  - Home: index.md
plugins:
  - techdocs-core

1. Start up Backstage
2. Register your catalog-info
3. Navigate to the Docs area
4. Then click on the Read Docs button for the repo you setup in the previous steps

Notice that you get a 404 error page suggesting that perhaps you are missing an index.md but if you look at the browser console there is a 404 error loading an index.html from an API route. Opening that route gets you to this error:

NotFoundError: Failed to read tree from https://ados-servername/organization/project/_git/repository, 404 Not Found at AzureUrlReader.readTree (webpack-internal:///…/backend-common/src/reading/AzureUrlReader.ts:106:15) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async UrlReaderPredicateMux.readTree (webpack-internal:///…/backend-common/src/reading/UrlReaderPredicateMux.ts:59:16) at async getDocFilesFromRepository (webpack-internal:///…/techdocs-common/src/helpers.ts:243:20) at async DocsBuilder.build (webpack-internal:///…/…/plugins/techdocs-backend/src/DocsBuilder/builder.ts:81:25) at async eval (webpack-internal:///…/…/plugins/techdocs-backend/src/service/router.ts:151:13)

Context

We are currently running Backstage using a fork, we’d really like to move away from that as it is a large amount of overhead to manage the merges. Getting this working would get us to that point.

Your Environment

The key element to this is using Azure DevOps Server 2020

[awanlin]

Hi @nadworny, this issue still happens for both TechDocs and the Scaffolder as they try to pull down the repo and the API URL is somewhat different for the on-prem version than what he cloud version uses. I previously tested this a few day ago as we are working on moving away from our fork as we are getting close to having a policy in place at work to contribute to open source projects.

This issue is unrelated to #6979 as this error will only come up using the Basic TechDocs setup but the workaround listed is for the Recommended setup where you are using Cloud storage, at least from my understanding.

If you are on discord I would suggest posting you issue there, they are super helpful!

[awanlin]

@nadworny I would love to but my work is going through the process of developing a policy for contributing to open source project like this so I need to wait for that to be in place first. I’ve tried my best to be very detailed here with the exact code changes needed and listed the lines and methods to be changed