Refresh catalog on every invoke (fresh_catalog: true)

Broad discussion here:

• https://github.com/meltano/meltano/discussions/6757

---

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2907

Originally created by @vischous on 2021-08-27 12:26:32

---

I want to have Meltano build a new catalog every run when running meltano invoke tap-oracle

A key at the SingerPlugin level probably makes sense, maybe call it fresh_catalog default to false?

More specific than #2850 as #2627 didn’t solve what I’m after. What I really want is a way to manage and watch my catalog change over time (#2677 / #2805 ), but this issue will be an incremental improvement over where I"m at today.

Today I delete the catalog and cache key from .meltano/run/tap-name/*

[tayloramurphy]

@aaronsteers that seems reasonable as a short-term fix. Long term I like the idea of meltano catalog - I think we can drive huge value around helping with the catalog and alerting on diffs prior to run execution. This is basically the “data contract” of the Singer world…

[cwegener]

Picking up this conversation after @edgarrmondragon pointed me to #2856 in Slack conversation.

Whilst providing an explicit catalog cache refresh via CLI (#2856) and via configuration setting (this issue) is a good medium term option, the important short-term enhancement is actually to address the point that @tayloramurphy pointed out, which is that the fact of a cached catalog file is currently not mentioned in the documentation.

I’ve had a quick look through the documentation, and the most sensible place to add a note about the cached catalog (and potentially a hint about using a tap reinstall as a workaround for the missing cache clear feature) seems to be the CLI reference documentation. e.g. in the select section.

Running select is how I had my first introduction to the catalog cache (using transferwise tap-postgres). And if I had chosen to start reading the documentation in order to find out why my select output does not match my list of source entities, I probably would have ended up reading the select CLI reference documentation.