Ignore setting values when selecting store location and store=auto

Given the following meltano.yml

default_environment: dev
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
    config:
      repository: meltano/meltano
environments:
- name: dev
- name: staging
- name: prod

if I run meltano --environment=prod config tap-github set repository "meltano/meltano meltano/hub"

I get

default_environment: dev
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
    config:
      repository: meltano/meltano meltano/hub
environments:
- name: dev
- name: staging
- name: prod

another scenario I tested:

default_environment: dev
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
environments:
- name: dev
- name: staging
- name: prod

meltano config tap-github set repository "meltano/meltano"

default_environment: dev
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-github
        config:
          repository: meltano/meltano
- name: staging
- name: prod

meltano --no-environment config tap-github set start_date "some_date"

default_environment: dev
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
    config:
      start_date: some_date
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-github
        config:
          repository: meltano/meltano
- name: staging
- name: prod

meltano --environment=prod config tap-github set start_date "some_other_date"

# UNEXPECTED RESULT
default_environment: dev
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
    config:
      start_date: some_other_date
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-github
        config:
          repository: meltano/meltano
- name: staging
- name: prod

meltano --environment=prod config tap-github set start_date "some_other_date" --store=meltano_environment

#EXPECTED RESULT
default_environment: dev
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
    config:
      start_date: some_other_date
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-github
        config:
          repository: meltano/meltano
- name: staging
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-github
        config:
          start_date: some_other_date

Updated Spec Proposal (AJ, 2022-06-14)

https://github.com/meltano/meltano/issues/6116#issuecomment-1155873908

[kgpayne]

…it made sense then to prefer overwriting the current value instead of adding a new possibly conflicting value in a difference place. That still feels right to me…

I think the Environments feature changes the assumption that the current store is likely the best, on the basis that Environments are specifically designed to override other stores. So I am generally in favour of:

i) Implementing a deterministic resolution of auto-store based on the setting type and current context (i.e Environment) only, as per AJ’s suggestion. ii) Improving logging messages to explain where auto-store placed a setting value and how to use --store if auto made a different choice to the users intent. iii) Adding visibility of setting values across all stores to the CLI config list command, to help users view and debug the resolution order of their settings in a particular context (especially for projects with config spread across multiple files).

[kgpayne]

@tayloramurphy

I assume that’s not part of your current PR, correct?

Correct 🙂 Created

• #6422

Are you thinking something like this:

Yep, I think that looks perfect. I also think it would be useful to show all the values in all the stores for an individual setting (including all environments etc.), especially for projects that have gone for the one-environment-per-subfile approach like Squared. But maybe that is a new detailed view with its own command path:

meltano config <plugin> list <setting>

A separate command would keep the summary list view concise whilst still giving users access to the full list if they want to find where a specific value is coming from.