🐛 Bug Report: @techdocs/cli - generating report flaky - sometimes no files generated

📜 Description

Sometimes (ca 10% of times) generating techdocs using currently latest version npx @techdocs/cli@v1.2.2 generate --verbose doesn’t generate any files, but command itself exits with exit code 0, indicating that generating files succeeded.

However last line from the output is:

info: Generating documentation…

See full output

npm WARN exec The following package was not found and will be installed: @techdocs/cli@1.0.0
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated keyv-memcache@1.3.3: This project has been renamed to @keyv/memcache which will be updated moving forward.
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
info: Using source dir /runner/_work/skeleton/skeleton
info: Will output generated files in /runner/_work/skeleton/skeleton/site
verbose: Creating output directory if it does not exist.
info: Generating documentation...

As a result our CI pipeline proceeds with publishing the techdocs using

npx @techdocs/cli@v1.2.2 publish \
            --publisher-type awsS3 \
            --storage-name our-backstage-techdocs-bucket \
            --entity default/component/skeleton

and as a result previously uploaded techdocs are unpublished from backstage:

npm WARN exec The following package was not found and will be installed: @techdocs/cli@1.0.0 info: Creating AWS S3 Bucket publisher for TechDocs info: Successfully connected to the AWS S3 bucket our-backstage-techdocs-bucket. info: Successfully uploaded all the generated files for Entity skeleton. Total number of files: 0 info: Successfully deleted stale files for Entity skeleton. Total number of files: 43

👍 Expected behavior

Generating techdocs should always succeed, not just most of the times! Output should end with following lines:

info: Generating documentation… info: Successfully generated docs from /runner/_work/skeleton/skeleton into /runner/_work/skeleton/skeleton/site using techdocs-container info: Done!

See full output when generating succeeds

npm WARN exec The following package was not found and will be installed: @techdocs/cli@1.2.2
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated keyv-memcache@1.3.3: This project has been renamed to @keyv/memcache which will be updated moving forward.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
info: Using source dir /runner/_work/skeleton/skeleton
info: Will output generated files in /runner/_work/skeleton/skeleton/site
verbose: Creating output directory if it does not exist.
info: Generating documentation...
info: Successfully generated docs from /runner/_work/skeleton/skeleton into /runner/_work/skeleton/skeleton/site using techdocs-container
info: Done!

Files generated when it succeeds

drwxr-xr-x  5 runner runner  4096 Nov 10 12:50 .
drwxr-xr-x 11 runner runner  4096 Nov 10 12:49 ..
-rw-r--r--  1 runner runner  9263 Nov 10 12:50 404.html
drwxr-xr-x  3 runner runner  4096 Nov 10 12:50 adrs
drwxr-xr-x  5 runner runner  4096 Nov 10 12:50 assets
-rw-r--r--  1 runner runner 10882 Nov 10 12:50 index.html
drwxr-xr-x  2 runner runner  4096 Nov 10 12:50 search
-rw-r--r--  1 runner runner   484 Nov 10 12:50 sitemap.xml
-rw-r--r--  1 runner runner   194 Nov 10 12:50 sitemap.xml.gz
-rw-r--r--  1 runner runner  1841 Nov 10 12:50 techdocs_metadata.json

👎 Actual Behavior with Screenshots

Actually sometimes there are no files generated into site folder: ls -la ./site:

drwxr-xr-x  2 runner runner 4096 Nov 10 11:31 .
drwxr-xr-x 11 runner runner 4096 Nov 10 11:31 ..

👟 Reproduction steps

• prepare techdocs
• save following test.sh script to the same folder where you have prepared techdocs:

#!/bin/bash
set -e # fail fast on error

# delete existing site folder
rm -rf site

# generate techdocs files
npx @techdocs/cli@v1.2.2 generate --verbose

echo "generated files in site folder:"
ls -la ./site

# fail if site folder is empty
if [ -z "$(ls -A ./site)" ]; then
  echo "ERROR: no files generated to site directory!"
  exit 1
fi

• allow executing that script using: chmod u+x ./test.sh
• run it in loop until it fails, for example using: watch --errexit ./test.sh (if you have watch installed - --errexit tells it to stop executing the script if it fails)

📃 Provide the context for the Bug.

When this issue happens, techdocs disappear from backstage (when publishing 0 files generated by TechDocs CLI)

🖥️ Your Environment

output from uname -a from our CI server (GitHub Actions runner) that executes it in CI pipeline:

Linux github-ci-runners-xfmsn-49x2s 5.15.0-1020-aws #24~20.04.1-Ubuntu SMP Fri Sep 2 15:29:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn’t find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

No response

[kgrafico]

Hi @atsu85 @agentbellnorm

I hope you are well, I have a problem with techDocs (v1.2.3). After login when I try to render a component I get this error:

Unable to get metadata for 'component:default/event-routes-db'; caused by Error: TechDocs metadata fetch failed; caused by Error: Unable to read stream; caused by CredentialsError: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1

In the logs I was able to see the same output and files generated when successfully generated as @atsu85.

I have noticed that if you try it several times it gets to work. Has any solution been found for this?

Thank you

[kgrafico]

@atsu85

This is the service that fails with a 404 when I try to access it.

Are you only getting the error when generating to publishing?