Unexpected behaviour when handling default values with Environments

What is the current bug behavior?

Meltano does not take Environment inheritance into account when setting a value to its default in an Environment.

What is the expected correct behavior?

Currently, when setting a setting value to it’s default, Meltano will unset all values (“to fall back to default”) rather than adding a new entry with the default value to meltano.yml. This behaviour was correct pre-environments, but does not take into account inheritance now that environments exist. The behaviour I would expect is that a new setting value would be added to the specific Environment with the default value, rather than the current behaviour that unsets values elsewhere in the inheritance chain.

Steps to reproduce

# add example tap
meltano add extractor tap-gitlab

# configure example setting with non-default value in base
meltano --no-environment config tap-gitlab set fetch_merge_request_commits true

# configure same setting with default value in an Environment
meltano environment=dev config tap-gitlab set fetch_merge_request_commits false

The above results in the removal of the True setting in base, rather than the addition of a False setting value in the environment dev.

Relevant logs and/or screenshots

Bug behaviour - removed True setting in plugin base configuration and no additional False setting in Environment dev. Expected behaviour:

true retained in base and false added to Environment dev (even though it happens to be the default value).

Possible fixes

Further regression test

Ensure we automatically catch similar issues in the future

• Write additional adequate test cases and submit test results
• Test results should be reviewed by a person from the team

/label ~bug /label ~“To Do”

[aaronsteers]

Agreed I think the simple path is best, to simply remove the equivalence check and always do a set when a set operation is requested.

If the user wants a clear, they should run meltano config ... unset.

I commented something similar in #6116.

Sidebar: when we have proper secrets support, equivalence checks may be impossible anyway. The behavior of “set when set is requested” should not require access to the prior/existing value.

[DouweM]

@tayloramurphy I’ve put this one into the current milestone assuming you agree that fixing config/environments related bugs is a good use of our time right now!