Support default branch syntax in GitHub UrlReader

Feature Suggestion

The GitHub discovery processor currently supports using - in place of the branch name to represent the default branch. For example: https://github.com/myorg/service-*/blob/-/catalog-info.yaml

The - is resolved into the actual name of the default branch by the discovery processor before it creates the child location in the catalog, which will then be used for reading the target YAML. This means that the entity location URL will in the end be set to for example https://github.com/myorg/service-foo/blob/master/catalog-info.yaml. This then triggers a conflict if you ever rename the default branch, since for example https://github.com/myorg/service-foo/blob/main/catalog-info.yaml is considered a different location. You then need to delete the original entity before the new location is picked up, as discussed in #9030.

I think we can solve this by keeping the - in place in the location URL, and instead power up GithubUrlReader.readUrl to support that syntax.

Possible Implementation

We think this is a good first issue and would be happy if anyone from the community wants to pick this up! ❤️

On the UrlReader side it should be enough to switch out this to not pass any ref query param if the ref is '-'.

In the discovery processor it should be enough to remove this logic and pass through the - in the URL instead. It is probably best to keep the check that makes sure that the repository has a default branch when using - as well.

These are pretty small changes, but we’d want to spend some time verifying that this all works as intended, allowing switching out the default branch and so on. One important thing to verify and possibly work around is to make sure that this doesn’t have a negative impact on existing catalogs that currently use GitHub discovery.

Context

#9030

[devth]

Sounds good! I’ll look into the git-url-parse issue later. I could wait till that’s resolved, or let me know if there’s help wanted on that too.

[Rugvip]

@devth good catch, thanks, updated 😁

And alright thank you for the heads up! Being able to properly parse the URL would be part of implementing this as well then. Can say that we’re aiming to move away from git-url-parse though (#9095)