Allow `python` interpreter overrides for specific plugins

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3411

Originally created by @aaronsteers on 2022-04-18 17:50:38

---

@edgarrmondragon suggested in https://gitlab.com/meltano/meltano/-/issues/2575#note_646844224 on Aug 9, 2021:

@aaronsteers @pandemicsyn Perhaps we can just add the --python option to the venv creation call if a new python field is not null in the plugin configuration.

In theory, it should be possible to create venvs that run plugins using a different Python interpreter than Meltano itself is using.

This could be useful when Meltano itself is installed on a newer version of Python (such as 3.10) at the time of this writing, but an older version may be needed to run a specific plugin. As we expand the number of installable plugins to include everything on https://hub.meltano.com, the cases where specific plugins need an older (or newer) version of Python will in theory become much more common.

Possible spec:

• Add an optional entry on plugin definitions: like python or (more explicit) python_interpreter_path.
• When not set, the python interpreter used would be the Meltano one.
• When set, the provided python interpreter would be used during plugin install (aka venv creation) instead of one Meltano is running under.

[aaronsteers]

I did some digging and found where I think the main code change would be:

https://github.com/meltano/meltano/blob/2052ede2c80fb0742da9809b8f461a3a6b28d662/src/meltano/core/venv_service.py#L222-L243

Perhaps an easier implementation vs dynamically accepting a Python version number would be to accept a path to a specific python interpreter. Functionally, this is basically the same thing, but it is more resilient and offers more predictable behavior overall, without Meltano having to be responsible for maintaining/governing the installation on the Python interpreters themselves.

    async def _pip_install(self, *pip_urls: str, upgrade: bool = False, python_path: Path | None = None):
        // ...
        args = [
            # use an override if provided or default to Meltano's:
            str(python_path or self.python_path),
            "-m",
            "pip",
            "install",
        ]
        // ...

And would just require adding support for one additional plugin config in yaml:

plugins:
  extractors:
  - name: tap-github
    pip_url: tap-github
    # use this for install instead of the Python interpreter Meltano is using:
    python_path: /usr/bin/python38

Or if we assume the interpreter is already on PATH, this version is much more portable:

plugins:
  extractors:
  - name: tap-github
     pip_url: tap-github
     python_path: python38

In and out of scope

What’s nice about this approach is that it is really simple to implement. But the cost of this simplicity is that in its first iteration, it probably should be a lever for users to implement as needed, rather than a robust automatically-delivered capability across taps and targets.

In other words: Meltano wouldn’t (yet) try to auto-select an interpreter for you based on known compatibility ranges - nor would Meltano try to understand what Python versions are installed on the local OS. But we would have docs that direct you to the troubleshooting steps, and we’d likely print helpful tips for Python cross-version compatibility+interop across versions whenever a plugin installation fails.

cc @EMCP, @tayloramurphy

[WillDaSilva]

@aaronsteers The approach you’ve outlined makes sense. Supporting both specifying a full path to the Python binary, and specifying a name to be found on $PATH (using shutil.which) is a nice touch. I’d give this a rough weight estimate of 4 - the most difficult part will surely be testing it.

To test it, a trick that might work (and would be easier than ensuring the environment has multiple distinct Python binaries) is to create a wrapper executable script around the Python binary that’s running the tests. When this wrapper script is run (because Meltano was told to use it for some plugin), it’ll leave behind some evidence of it having been run, which the test can then check for. This trick should work for both the full path and $PATH lookup.