feat: support container-based plugins

Problem to Solve

There is a whole world of plugins outside of the Python ecosystem that would benefit from having Meltano act as their foundation. Specific examples include:

• https://www.bytebase.com/ (written in Go)
• https://metriql.com/ (written in Kotlin)
• https://github.com/evidence-dev/evidence (written in Svelte)
• DuckDB (all sorts of languages going on)
• target-clickhouse: https://github.com/meltano/hub/issues/652

The current container_spec implementation is insufficient for some of the use cases of these plugins. There may be no pip_url and the user may likely never need or want to interact with the plugin outside of the containerized context.

The goal of this is to unlock a larger world of possible plugins for Meltano

Prior Art / Things to Consider

• We only currently support docker https://github.com/meltano/meltano/issues/3246
• https://github.com/meltano/meltano/issues/3181 should be considered as there are taps and targets that are written in other languages
• https://github.com/meltano/meltano/issues/3300 is a previous issue written by Pat while he was attempting to use Superset w/ Meltano
• https://github.com/meltano/meltano/issues/3127 will likely need to be considered as part of this
• https://github.com/meltano/meltano/issues/3246 Other container runtimes
• https://github.com/meltano/meltano/issues/3389 what we send into the container environment

Definition of Done

• Any dockerized tool should be able to be used within Meltano as a utility
  • Bytebase and Lightdash should be used as a test bed for validating.
• YAML spec is updated to support containerized plugins
• Consideration for other multiple container specs - docker can be used as primary for this
• Documentation updated in Meltano and the Hub to make it clear how to use these plugins

[tayloramurphy]

Can we agree though that the user experience (setting apart the experience of the maintainer) is identical regardless of whether container_spec inherits from plugin-level or is set explicitly per command? As we think about whether we have a full and complete dockerization/containerization featureset, I think it’s important to temporarily sideline berevity and focus on capability.

@aaronsteers I think that’s a fair statement. There are some oddities in the implementation that we can clear up likely in other issues.

One thing I do want to call out though is the perception challenge with container based plugins. The fact that it even feels like we might not support containerized plugins is itself a problem and we should fix through documentation and marketing. But also given recent conversations on priorities I believe we’re good to close this issue and keep the discussion in https://github.com/meltano/meltano/discussions/6459 and in specific docs issues.

[aaronsteers]

The current container_spec implementation is insufficient for some of the use cases of these plugins. There may be no pip_url and the user may likely never need or want to interact with the plugin outside of the containerized context.

@tayloramurphy and @DouweM - I’m not clear what the deficiency is, or if there is any deficiency at all. Non-Python plugins are already fully supported - as evidenced by the evidence.dev integration I built here: https://github.com/aaronsteers/aj-dataops-personal/blob/main/meltano.yml#L193-L246

Are we sure there are limitations or do we just need more ‘in the wild’ examples of container-based plugins to be added to the hub. If the former and we do have limitations, we should document those as issues to address.

1. This should be closed (I think) as I believe it’s already resolved with the current featureset: https://github.com/meltano/meltano/issues/3300
2. This is doable, for sure, but probably should not be prioritized, since won’t impact the drive for non-EL plugins: https://github.com/meltano/meltano/issues/3181
3. Non-docker runtimes is worth adding (https://github.com/meltano/meltano/issues/3246), but doesn’t change anything from an overall functionality/vision perspective. When we’re all feeling confortable that the current implementation is functionally complete, then I think we can certainly pick up the “other container runtimes” spec - which would bring the parity features to Ranger Desktop, containerd, etc. but would not otherwise change underlying behaviors.