[Techdocs] Bitbucket own hosted - commitsApiUrl , fileNameRegEx and other issues

Expected Behavior

Techdocs should get prepared, generated and published in local mode. Preparer stage should fetch the techdocs into ${inputDir}

Current Behavior

Failed to generate docs from ${inputDir} into ${outputDir} with error ${error.message} - with error undefined at TechdocsGenerator.run

Possible Solution

change https://github.com/backstage/backstage/blob/master/packages/backend-common/src/reading/BitbucketUrlReader.ts#L170 from const commitsApiUrl =${this.config.apiBaseUrl}/repositories/${project}/${repoName}/commits/${branch}; to const commitsApiUrl = ${this.config.apiBaseUrl}/projects/${project}/repos/${repoName}/branches/default; OR

const < condition to incorporate bitbucket.org and ownHosted > 
    ? `${this.config.apiBaseUrl}/repositories/${project}/${repoName}/commits/${branc`
    : `${this.config.apiBaseUrl}/projects/${project}/repos/${repoName}/branches/default`;

Checking condition - why the entry.autodrain(); is happening in below Line https://github.com/backstage/backstage/blob/master/packages/backend-common/src/reading/tree/ZipArchiveResponse.ts#L141

as the custom debug logs are showing that code is not going into below path

entry.pipe(fs.createWriteStream(path.join(dir, entryPath)));

Adding DEBUG to show exactly what is undefined error as thrown above -->

with error undefined at TechdocsGenerator.run

Steps to Reproduce

Step 1 set app config YAML as below

# Reference documentation http://backstage.io/docs/features/techdocs/configuration
techdocs:
  requestUrl: http://localhost:7000/api/techdocs
  storageUrl: http://localhost:7000/api/techdocs/static/docs
  builder: 'local' # Alternatives - 'external'
  generators:
    techdocs: 'local' # Alternatives - 'local'
  publisher:
    type: 'local' # Alternatives - 'googleGcs' or 'awsS3'. Read documentation for using alternatives.

Step 2 in your target repo ( user repo) add following annotations to catalog-info.yaml

  annotations:
    backstage.io/techdocs-ref: 'url:https://<mybitbuckethost.com>/projects/PRJ/repos/my-repo-1/browse'

add mkdocs.yml in project root ( target / user repo) as below

site_name: 'example-docs'

nav:
  - Home: index.md

plugins:
  - techdocs-core

create dir docs/ and add index.md with the following contents

# example docs

This is a basic example of documentation.

run backend and frontend in local

cd packages/backend 
yarn start 

cd ../..
yarn start 

Register component ( above repo : my-repo-1) using catalog-info YAML

click on docs… ( techdocs)

Context

Stacktrace / error logs

on frontend

Error: Failed to generate docs from C:\Users\Username\AppData\Local\Temp\backstage-MK--dzuVz2 into C:\Users\Username\AppData\Local\Temp\techdocs-tmp-MK-0Dik2s with error undefined at TechdocsGenerator.run (webpack-internal:///../techdocs-common/src/stages/generate/techdocs.ts:129:13) at runMicrotasks () at processTicksAndRejections (internal/process/task_queues.js:93:5) at async DocsBuilder.build (webpack-internal:///../../plugins/techdocs-backend/src/DocsBuilder/builder.ts:94:5) at async eval (webpack-internal:///../../plugins/techdocs-backend/src/service/router.ts:151:13)

DEBUG logs - custom - console.log

Console-logger: Printing contentDispositionHeader:--->>>> "attachment; filename=\"[e-mail]\"; filename*=UTF-8''my-repo-1-master%409b3e5ba59e9.zip" Console-logger: Printing archiveFileName:--->>>> "\"[e-mail]\";filename*=UTF-8''my-repo-1-master%409b3e5ba59e9"

backstage\packages\backend-common\src\reading\tree\ZipArchiveResponse.ts #L120 entry.autodrain();

it is not matching " if (entry.type === 'File' && this.shouldBeIncluded(entry)) " Therefore entry.autodrain();

and hence --> Size is 0 Console-logger: Printing archiveBitbucketResponse:--->>>> {"size":0,"timeout":0}

is above size value is byte size of the file?

and subsequently --> generator is not able to find the index.html/md or mkdocs.yml ( in the dir )

Your Environment

local windows OS

as well as linux container

( same behavior in both cases)

• NodeJS Version (v12):
• Operating System and Version (e.g. Ubuntu 14.04):
• Browser Information:

[OrkoHunter]

For the default branch error, can you switch to the branch in https://github.com/backstage/backstage/pull/4206 and see if that fixes it? 🤞

[OrkoHunter]

My recommendation on debugging is to test if the prepare step works. Generate step is bound to fail if there are no files available to generate upon. What I do is described here, Try to do a console.log(response.dir()) where response = await reader.readTree(URL) and I look into the content of the path printed, to see if the expected files were downloaded and extracted.