refactor: Move from `variant`+`name` to a unique `plugin_id`

Background

In the early days of Meltano, we managed all supported Meltano plugins in a single discovery.yml file. At that time, we had <40 taps, whereas this week (2022-11-28) we have 400+ unique plugins on the hub, with about a thousand total registrations variants.

The initial plugin_name was expanded to support a variant qualifier, which we defaulted to the organization or user name matching the owner ID of the repo in GitHub or GitLab.

Reasons to change

• A single plugin_id string or plugin_ref URL would simplify project maintenance. Plugins would still have name values in meltano.yml but there’d be a looser expectation around the name having unique significance outside of the meltano.yml file itself.

Proposal

In place of the current {plugin_name, variant}, adopt a new unique identifier such as plugin_id or just use the existing ref as the ID.

In place of…

plugins:
  extractors:
  - name: tap-github
    variant: meltanolabs
#...

We’d have:

plugins:
  extractors:
  - name: tap-github
    id: tap-github--meltanolabs
#...

Or perhaps better:

plugins:
  extractors:
  - name: tap-github
    ref: https://hub.meltano.com/meltano/api/v1/plugins/extractors/tap-github--meltanolabs
#...

Or we could let the local lock file name server as the ID, which itself could still reference back to the remote Meltano Hub URL ref:

plugins:
  extractors:
  - name: tap-github
    ref: plugins/extractors/tap-github--meltanolabs.lock
#...

Backwards compatibility

Backwards compatibility could be accomplished by making the new ID or Ref field optional, and when missing, calculating it on the fly as '{plugin_id}--{variant}' or 'meltano://{plugin_id}:{variant}' or similar.

---

Migrated from GitLab: https://gitlab.com/meltano/hub/-/issues/213

Originally created by @aaronsteers on 2022-03-11 17:28:32

Original thread

Wanted to log this question as related to the new tap-gitlab port. I created the new port as a fresh branch under the MeltanoLabs GitHub project. I’m not sure if I’ll be able to 100% match the prior behavior and at the same time, the tap can be “working” well for newer user cases.

The ideal delivery mechanism is probably to get reasonable amount of backwards compatibility but not 100%, and then publish as a variant name like meltanolabs-2 or similar. This would be a break from our current naming though, since the variant names today are expected to be exactly the same as the org name in GitHub or GitLab.

@tayloramurphy - Thoughts on this?

[edgarrmondragon]

@aaronsteers I like that plan. I’d only add

• Drop the --variant flag in meltano add would need to come between 2 and 3 above
• We don’t practice deprecation warnings too much, but should we emit warnings if variant is set in the plugin definition after 2 is implemented?

Also, the Hub is deeply coupled with variants since they’re used to group plugins for the same domain, e.g. tap-github. So, it may require a change to the data model to group them by domain instead. Wdyt?

[aaronsteers]

@edgarrmondragon and @tayloramurphy - I think there’s an order we could roll this out so that it would not need to be a breaking change.

This might not be the shortest path to resolution, but the below implementation path would (in theory) be incrementally deliverable.

1. Add internal plugin_id property for plugins in Meltano.
   1. plugin_id would be defined as a property using the pseudocode return self._plugin_id or f"{plugin_name}--{variant}". In other words, plugin_id would always be non-null and would default to our current naming convention for lock files and the Hub JSON API.
   2. The default path to the plugin lock file would become ${MELTANO_PROJECT_ROOT}/plugin/{plugin_id}.lock, which is equivalent to it’s current definition: ${MELTANO_PROJECT_ROOT}/plugin/{plugin_name}--{variant_name}.lock.
   3. For any plugin where plugin_id is specified explicitly, the plugin_id value will have precedence over the calculated default value ({plugin_name}--{variant_name}).
2. Update meltano add to ignore variant and instead register new plugins in meltano.yml using plugin_id.
   1. Whenever new plugins are added to meltano.yml via meltano add, the plugins will not get a variant key but instead they will get a plugin_id key.
      • E.g. Instead of adding a plugin with {name: tap-github, variant: meltanolabs}, the same plugin would be added using the identifiers {name: tap-github, plugin_id: tap-github--meltanolabs}.
   2. This requires that the step 1 above is already implemented, so that an explicit plugin_id value take precedence over the concatenation "{plugin_name}--{variant_name}".
3. Ensure plugin_name is purely cosmetic or referential in function.
   1. At this point, plugin_name should be more or less cosmetic at, at least for all plugins that have a plugin_id explicitly defined.
   2. As long as plugin_id is set, the user should be free to change plugin_name to whatever they want, without breaking the link to the lock file - except for two cases:
      1. Plugin name still needs to match the name as referenced in inherits_from relationships (if any). In other words, any inherits_from relationships will continue to inherit from sibling plugins via matching on plugin_name. This is important because inherits_from relationships inherit config as well as plugin_id, as well as other plugin-related descriptors defined in meltano.yml.
      2. Plugin name will still determine naming convention for environment variable auto-mapping to settings in the project. So, if I change the name from tap-github to tap-my-git-project, then this also changes the expected env var prefix from TAP_GITHUB_ to TAP_MY_GIT_PROJECT_.
4. Automatically handle name collisions. If ever a conflict arrises in plugin_name during meltano add, then initialize plugin_name with the value from plugin_id.
   1. Optionally, if plugin_id itself is already taken, use {plugin_id}_{n} as the name, where n is the lowest possible whole number that resolves the name collision.
   2. Since name is purely cosmetic and referential at this point, we can initialize name to any valid string, and we can allow users to then change name to whatever they want.