bug: Hard to debug valid JSON contents in generated catalog file

Meltano Version

1.79.0

Python Version

3.9

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

Python Docker image

Description

Following an upgrade from 1.76.0 to 1.79.0 I get the following error when trying to run once of our pipelines:

[2022-10-12 22:20:20,005] {pod_launcher.py:176} INFO - Event: tap-yodel-cat-target-snowflake.26bf63e794b442c7b35a66827cb0c95a had an event of type Running
[2022-10-12 22:20:20,018] {pod_launcher.py:136} INFO - meltano                       | Running extract & load...
[2022-10-12 22:20:20,018] {pod_launcher.py:136} INFO - meltano                       | Found state from 2022-10-12 20:00:11.154879.
[2022-10-12 22:20:21,612] {pod_launcher.py:136} INFO - meltano                       | ELT could not be completed: Cannot start extractor: Catalog discovery failed: invalid catalog: Expecting value: line 1 column 1 (char 0)

No further error is given. Not sure how to move past this or resolve the problem. I’m going to try upgrading to an even later version to try and solve the problem.

Code

No response

[edgarrmondragon]

Is this instead about an issue loading the existing catalog?

@alanmcruickshank Doesn’t look like. This line in the debug output suggests it’s generating the catalog:

Invoking: ['C:\\Users\\alanc\\dev\\tails-data-meltano\\meltano\\.meltano\\extractors\\tap-yodel\\venv\\Scripts\\tap-yodel', '--config', 'C:\\Users\\alanc\\dev\\tails-data-meltano\\meltano\\.meltano\\run\\tap-yodel\\tap.3c21d13d-81a8-42e8-a690-6776beb3e578.config.json', '--discover']

This make me think that the generated config is not right, and the tap depends on config to generate the catalog?

How would the output of --discover look like when using the config generated by Meltano:

pipenv run meltano invoke --dump=config tap-yodel > config.json
pipenv run tap-yodel --config config.json --discover

[DouweM]

@alanmcruickshank Can you try running with meltano --log-level=debug ... so we get some additional details?