integrations url processor doesn't implement support for private gitlab repos

Integrations for gitlab private repos do not work; only public repos can be pulled from, e.g. for catalog items.

Expected Behavior

The gitlab API (or different auth headers) need to be used with the token in order to get files from a private repo.

Current Behavior

Fails to load the referenced file.

Possible Solution

Use gitlab API endpoint to retrieve file content?

Steps to Reproduce

1. Make a private mirror of backstage/backstage.
2. Create an access token.
3. Modify example app-config.yaml (e.g. in app-config.local.yaml) to point a scaffolder entry at the private gitlab repo file. For example: the Documentation Template.
4. Run the demo server with yarn dev

[1] 2021-01-15T15:10:06.774Z catalog warn Failed item in location url:https://gitlab.cc.columbia.edu/cuit-ent-arch/backstage/blob/master/templates/docs-template/template.yaml, Error: YAML error, YAMLSemanticError: Implicit map keys need to be on a single line
[1]     at Module.generalError (webpack-internal:///../../plugins/catalog-backend/src/ingestion/processors/results.ts:59:56)
[1]     at parseEntityYaml (webpack-internal:///../../plugins/catalog-backend/src/ingestion/processors/util/parse.ts:45:66)
[1]     at parseEntityYaml.next (<anonymous>)
[1]     at UrlReaderProcessor.readLocation (webpack-internal:///../../plugins/catalog-backend/src/ingestion/processors/UrlReaderProcessor.ts:64:18)
[1]     at processTicksAndRejections (internal/process/task_queues.js:97:5)
[1]     at async LocationReaders.handleLocation (webpack-internal:///../../plugins/catalog-backend/src/ingestion/LocationReaders.ts:142:13)
[1]     at async LocationReaders.read (webpack-internal:///../../plugins/catalog-backend/src/ingestion/LocationReaders.ts:74:11)
[1]     at async HigherOrderOperations.refreshSingleLocation (webpack-internal:///../../plugins/catalog-backend/src/ingestion/HigherOrderOperations.ts:159:26)
[1]     at async HigherOrderOperations.refreshAllLocations (webpack-internal:///../../plugins/catalog-backend/src/ingestion/HigherOrderOperations.ts:138:9)
[1]     at async startRefresh (webpack-internal:///../../plugins/catalog-backend/src/util/runPeriodically.ts:42:9) type=plugin

Tracing the code shows https://github.com/backstage/backstage/blob/9d7b50f2f36f599df4c9ceacb18676fc0fdf72bc/packages/backend-common/src/reading/GitlabUrlReader.ts#L56 generates a GET with a PRIVATE-TOKEN header.

Gitlab only supports the PRIVATE-TOKEN header for the API, not file URLs.

Context

Trying to use backstage with private repos (on prem gitlab enterprise and/or github.com) and can’t reference individual git-hosted files with the url processor.

The current workaround is to make the repos public.

Your Environment

• NodeJS Version (v12): v12.20.0
• Operating System and Version (e.g. Ubuntu 14.04): MacOS Catalina 10.15.7 (19H114)
• Browser Information: N/A

[ruloweb]

@n2ygk all you have to do is to add a -/ in the URL xD, like this:

https://gitlab.cc.columbia.edu/cuit-ent-arch/backstage/-/blob/master/templates/docs-template/template.yaml

This is because of the logic in the getGitLabFileFetchUrl function, if the URL contains -/ it will use the API, otherwise, it will do a simple fetch.

https://github.com/backstage/backstage/blob/master/packages/integration/src/gitlab/core.ts#L41

It could be rewritten following the same logic as the github integration, what do you think @freben?.

[n2ygk]

Huh. scoped URLs. Go figure… Didn’t even know that was a thing. Aha! I see that our gitlab instance is on an old version (11.11.0) while newer versions will remove the legacy (no-dash) URLs so that will be a non-issue going forward.

Thanks @ruloweb!