Clarify "setting.aliases" behavior - in relation to previous "env_alias"

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3209

Originally created by @aaronsteers on 2022-02-02 21:42:27

---

As raised today (2022-02-02) in office hours, there is a benefit to env_aliases in that they can be used to ease migration across forks of the same tap or target. I wanted to propose, along the same vein, a setting_alias which would work as follows: We have settings.aliases capability which in theory could work at a higher abstraction level:

• The alias would be referenceable in meltano.yml config: blocks instead of the original name.
• The alias could be used in env variable injection so that TARGET_SNOWFLAKE_ACCOUNT could be used in place of TARGET_SNOWFLAKE_SNOWFLAKE_ACCOUNT.
• If both alias and the original name are set, this would be interpreted as a conflict. A hard failure would be raised: e.g.: The setting 'SNOWFLAKE_ACCOUNT' was provided along with its alias name 'ACCOUNT', which is not permitted. Please specify 'SNOWFLAKE_ACCOUNT' or 'ACCOUNT' but not both.

Caveats

• As we’re trying to move towards getting most/all config info from the hub (reducing dependency), this could be in conflict with that direction.
• Ideally, this would live in the hub definitions, rather than in discovery.yml, for the above reason.

Potential added benefit for user onboarding and switching between variants

There could be added benefit to aliases, in that certain cross-implementation representations could be implemented by Meltano (and other orchestrators) generically. Many taps have common config settings but are each subject to variations in naming.

Example config options which would make good candidates for generic and cross-implementation aliases:

• account
• username
• password
• port
• auth_token
• refresh_token
• client_secret
• host
• s3_bucket
• hard_delete
• target_schema_name
• target_db_name

If we allowed the Hub entry to declare any of these aliases/forms of settings in the hub metadata, then orchestrators could leverage this info, and also (back to one of the original benefits of env_aliases), users could swap between variants without having to fully rewrite their config - at least not for those core elements that variants would have in common.

Benefits for dbt abstraction

Downstream tools like dbt generally need a small number of variables related to where the data will land. Having common aliases for these functionally-significant parameters would allow Meltano to inject values and/or read values from those important settings. So, for instance, Meltano could know which setting names have the aliases target_schema_name and target_db_name, then we can use that to pass the values along to dbt for identifying the source of a transformation.

---

Update 2022-06-03

The scope of this is to clarify the behavior with setting aliases. Discussed in the GitLab issue we decided on:

• users can set config via the setting name and any of the values in the setting aliases array
• we would hard fail if multiple are set, but accept any of them
• environment variable - accept any of the options, fail if mutliple are present
  • PLUGIN_NAME_SETTING_NAME
  • PLUGIN_NAME_SETTING_ALIAS_1
  • PLUGIN_NAME_SETTING_ALIAS_2

[cjohnhanson]

@tayloramurphy thanks, that totally clears things up.

[DouweM]

@cjohnhanson To be clear, I think the behaviors of meltano config, meltano config set and meltano config list in https://github.com/meltano/meltano/issues/3135#issuecomment-1149247582 are bugs, but I don’t actually care that much about exposing the aliases in the environment as I referenced in https://github.com/meltano/meltano/issues/3135#issuecomment-1149235097. It’s more clear to the user if the aliases are only respected on the input side (when the user provides config to Meltano), not the output (when Meltano passes config to the plugin).

Defer to @tayloramurphy for the actual spec, though.