docs: Variable Inheritance unclear at plugin vs env level

Meltano Version

2.4.0

Python Version

3.8

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

Ubuntu

Description

https://github.com/meltano/meltano/issues/5983 and the demo that was done looks interesting. Tried to read docs on this and it looks like there are not any or am I missing it?

To reproduce

meltano init variable_inherit
meltano add extractor tap-csv
meltano add loader target-jsonl
meltano config tap-csv set testvar inheirt_me
meltano config target-jsonl set please_inheirt ${TAP_CSV_TESTVAR}
meltano config target-jsonl list

Output of last comand

2022-08-10T17:03:11.179400Z [info     ] Environment 'dev' is active
2022-08-10T17:03:11.688732Z [info     ] The default environment (dev) will be ignored for `meltano config`. To configure a specific Environment, please use option `--environment=<environment name>`.
destination_path [env: TARGET_JSONL_DESTINATION_PATH] current value: 'output' (default)
        Destination Path: Sets the destination path the JSONL files are written to, relative
to the project root.

The directory needs to exist already, it will not be created
automatically.

To write JSONL files to the project root, set an empty string (`""`).

do_timestamp_file [env: TARGET_JSONL_DO_TIMESTAMP_FILE] current value: False (default)
        Include Timestamp in File Names: Specifies if the files should get timestamped.

By default, the resulting file will not have a timestamp in the file name (i.e. `exchange_rate.jsonl`).

If this option gets set to `true`, the resulting file will have a timestamp associated with it (i.e. `exchange_rate-{timestamp}.jsonl`).


Custom, possibly unsupported by the plugin:
please_inheirt [env: TARGET_JSONL_PLEASE_INHEIRT] current value: None (from `meltano.yml`: '${TAP_CSV_TESTVAR}')

To learn more about loader 'target-jsonl' and its settings, visit https://hub.meltano.com/loaders/target-jsonl

Code

No response

[tayloramurphy]

I’ll consider the scope of this issue to be to update the docs to clarify the above two points.

@cjohnhanson I’ll update the title to reflect that. thanks for the review 👍

[cjohnhanson]

@tayloramurphy (CC: @visch) – everything you said is correct. If you think of the various env keys as a tree with the root-level env key in meltano.yml as its root node and edges between nodes representing value inheritance, values are inherited “vertically” by children nodes from parent nodes, but not inherited “horizontally” between nodes at the same level of the tree. I think a simple diagram representing this would probably be the best way to clarify this in the docs–it’s difficult to describe clearly and concisely with words alone.

I think another potential docs limitation that should be tackled here is to disambiguate “environment variables” from “settings”. In practice, we generally pass settings values to plugins as environment variables but they’re very distinct within the codebase. The full inheritance “tree” is only evaluated right at invocation time by the plugin invoker, since the final resolution/expansion of env vars depends on values that aren’t available at any single settings layer. So in short, yes, plugin settings configured via meltano config are settings and are managed differently than environment variables defined via env keys.

I’ll consider the scope of this issue to be to update the docs to clarify the above two points.