Python-based Plugin Architecture

I think we have a path forward on python plugin architecture (https://gitlab.com/groups/meltano/-/epics/134#note_981098666) which does not require modifying Meltano itself (at least not in the first iterations).

Since pip_url for any repo can include multiple libraries, we can extend any plugins’ capabilities by adding a meltano-managed python tool to any plugin that needs capabilities. Then those capabilities just need to registered as CLI-executable and be mapped to commands and capabilities in the Meltano plugin definition.

Example: registering dbt packages in packages.yml

Today we have custom code in Meltano that adds entries to packages.yml for dbt during meltano add transform .... We could replace it as follows:

1. Create a new python tool called meltano-dbt-helpers with a registered CLI command meltano-dbt-util add package.
2. Add that package to the pipurl for all dbt plugins. E.g. pip_url: dbt-snowflake meltano-dbt-helpers.
3. Register commands in the package that leverage the helper. E.g.

   commands:
    add-dep:
      command: meltano-dbt-util
      args: add package
   

Then from the CLI as a user:

meltano run dbt:add-dep dbt-util

Would add dbt-util to the package.

For “smarter” integrations, we just need a contract that pairs capabilities like service with expected commands, like ui, up, and down.

Comparison with other approaches

The best part of this approach is that it doesn’t require any code changes in Meltano - we and community members could implement this today with existing features.

[tayloramurphy]

@pandemicsyn per your feedback in https://github.com/meltano/meltano/issues/6398 I’m assigning to you for the current iteration.

[pandemicsyn]

@tayloramurphy so for the sdk we currently have:

• the repo: http://github.com/meltano/sdk
• the package name: singer-sdk & singer_sdk

So for this new addition, maybe something like:

• the repo: http://github.com/meltano/edk
• the package name: meltano-edk & meltano_edk