question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

feature request: `config test` command should return more helpful logs

See original GitHub issue

I’m not sure what options are available but this is related to a slack thread https://meltano.slack.com/archives/CMN8HELB0/p1670907167441139 where meltano config tap-google-analytics test was failing and it was hard to figure out what was wrong.

In this case the view_id was expected to be a string but was being passed in as an integer. The error message just says Plugin configuration is invalid with no other context. Eventually we ran the tap with invoke and it showed log messages that explained the misconfiguration singer_sdk.exceptions.ConfigValidationError: Config validation failed: <number> is not of type 'string'. I also got the same generic Plugin configuration is invalid message when testing a purposefully wrong client secret.

Potentially Helpful Ideas:

  1. Try to re-log the exception. Maybe we can say exactly what part of the json schema didnt pass validation or if the tap is executed we can re-log the exception (although I know thats tricky).
  2. Add a log to the config/test error output about what it means. I.e. “Plugin configuration is invalid. This could be related to a schema violation or because the tap was unable to connect to the source system with provided configs.” 🤷
  3. Add a suggestion to try using invoke to see if the tap returns a helpful log about whats misconfigured
  4. The hub tells users to run config/test. Maybe it should also have a note to try running invoke.

cc @menzenski

Issue Analytics

  • State:open
  • Created 9 months ago
  • Reactions:2
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

3reactions
aaronsteerscommented, Dec 13, 2022

Following from the roadmap outlined here:

I’ve logged the ‘first step’ here:

If that is built, then:

  1. The Singer logs would be logged into a file during discovery.
  2. If the task fails, we’d print the path to the log file, which you could open and view logs.
  3. Logs would describe the validation error.
2reactions
menzenskicommented, Dec 13, 2022

It’s perhaps worth noting that if the --log-level=debug parameter is passed (meltano --log-level=debug config tap-google-analytics test), the same clear error message is written to the logs but it’s effectively buried in many other log messages.

2022-12-13T15:41:55.328190Z [debug    ] File "/Users/matt/dev/pudl/src/.meltano/extractors/tap-google-analytics/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 257, in _validate_config
2022-12-13T15:41:55.328302Z [debug    ] raise ConfigValidationError(summary)
2022-12-13T15:41:55.328374Z [debug    ] singer_sdk.exceptions.ConfigValidationError: Config validation failed: XXXXXXXXX is not of type 'string'
2022-12-13T15:41:55.328452Z [debug    ] JSONSchema was: {'type': 'object', 'properties': {'start_date': {'type': ['string'], 'format': 'date-time', 'description': 'The earliest record date to sync'}, 'view_id': {'type': ['string'], 'description': 'Google Analytics View ID'}, 'client_secrets': {'type': ['object', 'null'], 'properties': {}, 'description': 'Google Analytics Client Secrets Dictionary'}, 'key_file_location': {'type': ['string', 'null'], 'description': 'File Path to Google Analytics Client Secrets'}, 'oauth_credentials': {'type': ['object', 'null'], 'properties': {'access_token': {'type': ['string', 'null'], 'description': 'Google Analytics Access Token'}, 'refresh_token': {'type': ['string', 'null'], 'description': 'Google Analytics Refresh Token'}, 'client_id': {'type': ['string', 'null'], 'description': 'Google Analytics Client ID'}, 'client_secret': {'type': ['string', 'null'], 'description': 'Google Analytics Client Secret'}}, 'description': 'Google Analytics OAuth Credentials'}, 'reports': {'type': ['string', 'null'], 'description': 'Google Analytics Reports Definition'}, 'end_date': {'type': ['string', 'null'], 'description': 'The last record date to sync'}, 'stream_maps': {'type': ['object', 'null'], 'properties': {}, 'description': 'Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html).'}, 'stream_map_config': {'type': ['object', 'null'], 'properties': {}, 'description': 'User-defined config values to be used within map expressions.'}, 'flattening_enabled': {'type': ['boolean', 'null'], 'description': "'True' to enable schema flattening and automatically expand nested properties."}, 'flattening_max_depth': {'type': ['integer', 'null'], 'description': 'The max depth to flatten schemas.'}}, 'required': ['start_date', 'view_id']}
2022-12-13T15:41:55.414234Z [debug    ] Deleted configuration at /Users/matt/dev/pudl/src/.meltano/run/tap-google-analytics/tap.b8c6ff32-c496-4377-a255-ee76580171dd.config.json
2022-12-13T15:41:55.415769Z [debug    ] Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
Read more comments on GitHub >

github_iconTop Results From Across the Web

Unit test reports - GitLab Docs
You configure the Unit test reports and immediately GitLab collects and exposes them in the merge request. No more searching in the job...
Read more >
logging.config — Logging configuration — Python 3.11.1 ...
Logging configurations will be sent as a file suitable for processing by dictConfig() or fileConfig() . Returns a Thread instance on which you...
Read more >
Advanced Git Log | Atlassian Git Tutorial
Formatting the output of git log is essential when working with large repositories. In this tutorial, we'll show you how to do it...
Read more >
Best Practices - Cypress Documentation
Organizing Tests, Logging In, Controlling State; Selecting Elements; Assigning Return Values; Visiting external sites; Having tests rely on the state of ...
Read more >
Cli Commands - Icinga 2
These CLI commands will allow you to use certain functionality provided by and around Icinga 2. Each CLI command provides its own help...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found