bug: Meltano not seeing config in yml files other than `meltano.yml`

Meltano Version

2.8.0

Python Version

3.10

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

macOS Monterey 12.3

Description

I’m developing a tap and have configuration split across meltano.yml and another file in the same directory called private.meltano.yml. Meltano appears to see tap config in the former file and environment variables but not the latter file. The config below results in Meltano throwing the error Config validation failed: 'page_ids' is a required property and meltano config tap-facebook-pages list confirms that the page_ids config option is null.

# meltano.yml

version: 1
send_anonymous_usage_stats: true
project_id: "tap-facebook-pages"
include_paths:
  - ./*.meltano.yml
default_environment: test
environments:
- name: test
plugins:
  extractors:
  - name: "tap-facebook-pages"
    namespace: "tap_facebook_pages"
    pip_url: -e .
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    config:
      start_date: '2010-01-01T00:00:00Z'
    settings:
    - name: user_token
      kind: password
    - name: page_ids
    - name: start_date
      value: '2022-10-01T00:00:00Z'
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

# private.meltano.yml

environments:
- name: test
  config:
    plugins:
      extractors:
      - name: "tap-facebook-pages"
        config:
          page_ids:
          - 'xxxxxxxxxxxx'

I tried the alternatives suggested here in Slack and saw the same outcome/error:

• making plugins the root key of private.meltano.yml
• removing any reference to environments from either file
• removing environments from just meltano.yml but keeping it in private.meltano.yml

Code

❯ meltano elt tap-facebook-pages target-jsonl                  
2022-11-08T14:21:32.409844Z [info     ] Running extract & load...      name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl
2022-11-08T14:21:32.456508Z [warning  ] No state was found, complete import.
2022-11-08T14:21:33.316046Z [info     ] ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/bin/tap-facebook-pages', '--config', '/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/run/elt/2022-11-08T142131--tap-facebook-pages--target-jsonl/4fb85bf6-30d7-4772-9cda-62f5e6b9ed56/tap.ed63a9c0-c313-494b-9039-d060dbc0daa1.config.json', '--discover'] returned 1 with stderr: cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.316221Z [info     ]  Config validation failed: 'page_ids' is a required property cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.316287Z [info     ] JSONSchema was: {'type': 'object', 'properties': {'user_token': {'type': ['string'], 'description': 'Long-lived user access token with access to all pages.', 'secret': True, 'writeOnly': True}, 'page_ids': {'type': 'array', 'items': {'type': ['string']}, 'description': 'Page IDs of Facebook pages for which to fetch data.'}, 'start_date': {'type': ['string', 'null'], 'format': 'date-time', 'description': 'The earliest record date to sync.'}, 'stream_maps': {'type': ['object', 'null'], 'properties': {}, 'description': 'Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html).'}, 'stream_map_config': {'type': ['object', 'null'], 'properties': {}, 'description': 'User-defined config values to be used within map expressions.'}, 'flattening_enabled': {'type': ['boolean', 'null'], 'description': "'True' to enable schema flattening and automatically expand nested properties."}, 'flattening_max_depth': {'type': ['integer', 'null'], 'description': 'The max depth to flatten schemas.'}}, 'required': ['user_token', 'page_ids']} cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.316354Z [info     ] Traceback (most recent call last): cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.316406Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/bin/tap-facebook-pages", line 8, in <module> cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.317080Z [info     ]     sys.exit(TapFacebookPages.cli()) cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.317154Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.317218Z [info     ]     return self.main(*args, **kwargs) cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.317300Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.317357Z [info     ]     rv = self.invoke(ctx)      cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.320293Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.320344Z [info     ]     return ctx.invoke(self.callback, **ctx.params) cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.320391Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.323294Z [info     ]     return __callback(*args, **kwargs) cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.323364Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 489, in cli cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.325282Z [info     ]     tap = cls(  # type: ignore  # Ignore 'type not callable' cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.325333Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 95, in __init__ cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.326060Z [info     ]     self.mapper.register_raw_streams_from_catalog(self.catalog) cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.326112Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 157, in catalog cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.326172Z [info     ]     self._catalog = self.input_catalog or self._singer_catalog cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.328011Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 249, in _singer_catalog cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.328075Z [info     ]     for stream in self.streams.values() cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.328686Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 120, in streams cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.328773Z [info     ]     for stream in self.load_streams(): cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.328828Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 281, in load_streams cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.330726Z [info     ]     for stream in self.discover_streams(): cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.330800Z [info     ]   File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/tap_facebook_pages/tap.py", line 86, in discover_streams cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.331845Z [info     ]     page_id: self.exchange_token(page_id) for page_id in self.config["page_ids"] cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.331922Z [info     ] KeyError: 'page_ids'           cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.331976Z [info     ] .                              cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.333463Z [info     ] For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag. cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.333538Z [info     ] Note that you can also check the generated log file at '/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/logs/elt/2022-11-08T142131--tap-facebook-pages--target-jsonl/4fb85bf6-30d7-4772-9cda-62f5e6b9ed56/elt.log'. cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
2022-11-08T14:21:33.334551Z [info     ] For more information on debugging and logging: https://docs.meltano.com/reference/command-line-interface#debugging cmd_type=elt name=meltano run_id=4fb85bf6-30d7-4772-9cda-62f5e6b9ed56 state_id=2022-11-08T142131--tap-facebook-pages--target-jsonl stdio=stderr
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/bin/tap-facebook-pages', '--config', '/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/run/elt/2022-11-08T142131--tap-facebook-pages--target-jsonl/4fb85bf6-30d7-4772-9cda-62f5e6b9ed56/tap.ed63a9c0-c313-494b-9039-d060dbc0daa1.config.json', '--discover'] returned 1 with stderr:
 Config validation failed: 'page_ids' is a required property
JSONSchema was: {'type': 'object', 'properties': {'user_token': {'type': ['string'], 'description': 'Long-lived user access token with access to all pages.', 'secret': True, 'writeOnly': True}, 'page_ids': {'type': 'array', 'items': {'type': ['string']}, 'description': 'Page IDs of Facebook pages for which to fetch data.'}, 'start_date': {'type': ['string', 'null'], 'format': 'date-time', 'description': 'The earliest record date to sync.'}, 'stream_maps': {'type': ['object', 'null'], 'properties': {}, 'description': 'Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html).'}, 'stream_map_config': {'type': ['object', 'null'], 'properties': {}, 'description': 'User-defined config values to be used within map expressions.'}, 'flattening_enabled': {'type': ['boolean', 'null'], 'description': "'True' to enable schema flattening and automatically expand nested properties."}, 'flattening_max_depth': {'type': ['integer', 'null'], 'description': 'The max depth to flatten schemas.'}}, 'required': ['user_token', 'page_ids']}
Traceback (most recent call last):
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/bin/tap-facebook-pages", line 8, in <module>
    sys.exit(TapFacebookPages.cli())
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 489, in cli
    tap = cls(  # type: ignore  # Ignore 'type not callable'
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 95, in __init__
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 157, in catalog
    self._catalog = self.input_catalog or self._singer_catalog
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 249, in _singer_catalog
    for stream in self.streams.values()
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 120, in streams
    for stream in self.load_streams():
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/.meltano/extractors/tap-facebook-pages/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 281, in load_streams
    for stream in self.discover_streams():
  File "/Users/prratek.ramchandani/projects/singer_taps/tap-facebook-pages/tap_facebook_pages/tap.py", line 86, in discover_streams
    page_id: self.exchange_token(page_id) for page_id in self.config["page_ids"]
KeyError: 'page_ids'

[tayloramurphy]

@edgarrmondragon @aaronsteers I’d like us to see if we can get this figured out. It’s possible this is something with meltano elt specifically because when I split the config locally and run meltano config <tap> it’s able to pick up the config spread across multiple files.

[prratek]

Confirmed that this fix works. Thanks!