Nested config values in setting arrays and objects don't have environment variables expanded

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3245

Originally created by @alxthm on 2022-02-15 09:06:57

---

Related to #2366

What is the current bug behavior?

I was having trouble using environment variables in the meltano.yml file, and it seems they are only supported for top-level configuration options.

If I define SOME_ENV_VAR=SOME_VALUE in the .env file and I write this configuration yaml:

plugins:
  extractors:
    - name: tap-spreadsheets-anywhere
      variant: ets
      pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
      config:
        test: $SOME_ENV_VAR
        tables:
          - path: $SOME_ENV_VAR
            name: my_table

Then this is the actual configuration used by meltano:

$ meltano config tap-spreadsheets-anywhere
{
  "tables": [
    {
      "path": "$SOME_ENV_VAR",
      "name": "my_table"
    }
  ],
  "test": "SOME_VALUE"
}

And the tables.path variable is not set.

What is the expected correct behavior?

The environment variable SOME_ENV_VAR should expend to SOME_VALUE in the tables array as well, and we should have:

$ meltano config tap-spreadsheets-anywhere
{
  "tables": [
    {
      "path": "SOME_VALUE",
      "name": "my_table"
    }
  ],
  "test": "SOME_VALUE"
}

Steps to reproduce

• add the configuration above in meltano.yml
• add the line SOME_ENV_VAR=SOME_VALUE to your .env file
• see the result with meltano config tap-spreadsheets-anywhere

Relevant logs and/or screenshots

Please use code blocks (```) to format console output

Possible fixes

If you can, link to the line of code that might be responsible for the problem or suggest a fix

Further regression test

Ensure we automatically catch similar issues in the future

• Write additional adequate test cases and submit test results
• Test results should be reviewed by a person from the team

[pnadolny13]

@tayloramurphy bumping here related to another user needing this https://github.com/meltano/meltano/issues/3435#issuecomment-1333985077.

[edgarrmondragon]

@aaronsteers object settings can be worked around by using the dot syntax, e.g. my.nested.setting and can expand environment variables that way. For example: https://github.com/meltano/sdk/issues/1073