Update terminal environment variable precedence to be in line with currently xfail-ing tests

We want environment variable resolution to behave as described in this miro board.

This behavior is described in test cases written in tests/meltano/core/test_environment_variables.pywhich which currently have xfail=True. Behavior should be updated so that these tests pass. The relevant test cases will have a comment referencing this issue.

Related:

• Exploratory issue:
  • https://github.com/meltano/meltano/issues/3100
  • https://github.com/meltano/meltano/pull/5938
• Also found:
  • https://github.com/meltano/meltano/issues/5983
• Xfail (expected-to-fail) tests to fix: https://github.com/meltano/meltano/commit/17ce6c65d2ea3f4b5851528ec15de45e22843be8

[cjohnhanson]

@tayloramurphy yes exactly, I failed to disambiguate “environment variables” from “plugin settings” since I think in practice it should be rare to pass to plugins environment variables which are not also configuration settings (or maybe it’s not rare and I’m further displaying my ignorance here 😅). The existing unit tests do test for “environment variable precedence with regard to the env at all levels” and that precedence behavior works as expected.

[cjohnhanson]

@aaronsteers (CC @kgpayne--you might have also come to this conclusion or otherwise have some insight here)

After banging my head against this for a long time, I think I’ve come to the embarrassing conclusion that current behavior is in line with the spec and I misinterpreted the precedence order when writing these tests.

Basically, the currently failing tests fail because they have environment variables that are set at the terminal or active environment level and in the plugin- and/or environment-plugin level. I interpreted the precedence order in the spec to indicate that the plugin- or environment-plugin level value should “win”, but these tests fail because the values from the terminal or active environment context are “winning” and taking precedence over the plugin- and environment-plugin level values. My mistake in these tests is that the environment variables being tested are also valid settings for the mock plugin. This means that the values set for them are resolved into the plugin settings/config and then those resolved settings are mapped to environment variables, which take precedence over the plugin- and environment-plugin level envs.

The key takeaway here is that environment variables in terminal or active environment contexts will always take precedence over plugin or environment-plugin values for those same variables if those env variables in question are valid keys for a plugin setting. My reading of the spec was that plugin or environment-plugin should “win” over terminal or active environment context, but I think this was incorrect and the existing behavior is actually correct.

I think I particularly owe @kgpayne a mea culpa for potentially sending him down an unnecessary rabbit hole in trying to resolve this issue.

I’m going to update the tests to reflect my new reading of the spec and then we can finally close out this issue. FYI @tayloramurphy if you have any additional thoughts on this.