Add `--interactive --all` CLI flags in `meltano config set`

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3227

Originally created by @aaronsteers on 2022-02-09 21:01:33

---

Based on recent discussions around the difficulties in knowing which settings to configure when adding a new plugin, we should add an interactive mode for setting all setting for a plugin, or any specific settings value.

Something like…

> meltano config dbt-snowflake set --interactive --all

You are now configuring the "dbt-snowflake" transformer plugin.
For help, please refer to plugin documentation: https://...

We will now step through each setting. Leave any entry blank to skip, 
or enter a new value. Password values will automatically be hidden 
from display for any settings of type 'password'.

Press Ctrl+C to cancel at any time.

Setting #1: project_dir
Description: The root dbt path.
Type: string
Default value: $MELTANO_PROJECT_ROOT/transform
Current value: (none)
New Value: _ 

(skipped)

....

Setting #7: snowflake_user
Description: The username to use in Snowflake authentication.
Type: string
Default value: (none)
Current value: (none)
New Value: snowflake_prod_user

Value saved successfully to meltano.yml!

Setting #8: snowflake_password
Description: The password to use in Snowflake authentication.
Type: password
Default value: $MELTANO_PROJECT_ROOT/transform
Current value: (none)
New Value: ********** 

Value saved successfully to Meltano System DB!

Spec Details

• Interactive behavior is triggered bia the --interactive flag in meltano config ... set.
• Interactive behavior can be triggered for a single setting or with --all to set all settings associated with the given plugin.
• The --all option requires the --interactive flag. If --all is specified without --interactive, Meltano should fail.
• After each setting is set, Meltano should respond with Value saved successfully to _<store>_!
  • The store could be .env, the SystemDB, or to Meltano.yml.
  • Confirming and saving after each setting is set gives real-time feedback to the user about changes they have applied, and allows the remaining operation to be canceled without loss of any setting values already saved.
• Individual settings should be skippable.
• Password fields should either print asterisks or simply not echo anything while users are typing.
• The new setting value should not be provided inline at the command line if --interactive is set. So, passing the setting value and also passing --interactive should cause Meltano to fail.

Ability to “skip” settings

Skipping a setting may not be intuitive if done inline with the setting value input.

Setting #1: project_dir
Description: The root dbt path.
Type: string
Default value: $MELTANO_PROJECT_ROOT/transform
Current value: (none)
Would you like to set a new value? [Y/n]: n

Skipped.

Setting #2: snowflake_user
Description: The username to use in Snowflake authentication.
Type: string
Default value: (none)
Current value: (none)
Would you like to set a new value? [Y/n]: Y
New Value? []: snowflake_prod_user

Value saved successfully to meltano.yml!

...

Without a separate prompt to skip or set the value, the user would generally have to type something like an empty string or s for skip, or similar. However, those are potentially valid answers to the new value prompt.

Phases

• The initial merge should at minimum provide --all --interactive support.
• Optionally, setting only one value (--interactive without --all) can be included in initial scope or added in a subsequent iteration.
  • When setting only a single value, presumably we do not need to allow the “skip” option described above, since the user can simply abort (ctrl+c) to cancel if they do not want the value set.
• Not required in this first iteration, but a future iteration should require users to first define a setting’s type if the type is not yet set. Would be a path for resolving #2768. For example, if the user runs meltano config tap-snowflake set --interactive snowflake_password, that should require the user to first define the setting kind (kind is password in this case), which then drives correct parsing and storage of the provided value.

[tayloramurphy]

@kgpayne option 2 is a good iteration and I like AJ’s comments in https://github.com/meltano/meltano/issues/2971#issuecomment-1212326558 on having it behind a feature flag and configurable.

I can open a new issue on the possibility of a short_description key for settings. Note that the usage key is for the entire plugin and not a specific setting.

---

We also discussed printing a URL to the homepage/repo with a prompt along the lines of “for detailed help, see the plugins’ documentation” to compliment a shortened description.

We already print the link at least in your screenshot, so updating the text there should be good too!

[aaronsteers]

@kgpayne - Yes, I agree option 2 is great for this phase.

We want to be careful that the addition of rich doesn’t automatically change other behaviors, but once the feature flag is in place for #2971, that should be safe to add.

Also - thanks for the rendering example. I think it looks nice, and it is worthwhile having markdown-rendering-to-terminal as a tool in our toolbox overall.