Add an option for pip to manage a different "environment"

What’s the problem this feature will solve?

At the moment, pip can only fully[^1] manage the current Python environment. This makes it hard to have environments without pip installed (for example, when preparing a zipapp, or an embedded Python installation).

Being able to specify a target “environment” (in the sense of a set of paths like sysconfig scheme) which pip can manage, would allow a single shared pip to be used for multiple Python environments.

[^1]: The --target, --root and --prefix options only apply to pip install.

Describe the solution you’d like

Add a global option to pip that sets the “environment” to be managed. The UI for this option is yet to be decided - in the most explicit case, values are needed for the 5 sysconfig paths that pip can install data into, but shortcuts for common cases would be important for a good user experience.

All pip commands will use the specified environment.

Alternative Solutions

• The existing --target, --root and --prefix options all attempt to handle this, but they lack support for upgrades and uninstalls, as well as not being integrated with informational commands like list.
• Running pip with the Python environment’s python executable works, but only when the target environment is a virtual environment. This does not cover cases like preparing a zipapp, or installing into the library area of an app that embeds Python.
• Rather than having a UI for selecting a scheme, it would be possible to have a (simple) programmatic API to allow people to set a scheme, and then invoke pip[^1]. But that would make “manage another environment” inaccessible from the fundamental python -m pip command.

[^1]: My tests suggest that this is possible right now, if you’re willing to make some pretty horrible hacks to pip’s internals in your script.

Additional context

This is closely related to #4575. A lot of the discussion on that issue covered handling environments that “shadow” each other (user shadowing system, and --use-system-site virtual environments). This issue explicitly avoids that problem, focusing on the case of a single standalone environment, with shadowing being left as an issue for how the user sets up their sys.path.

This proposal doesn’t stand a chance of working with the “legacy” direct setup.py install method (for editables or normal installs), as that method delegates installation to setuptools, and it is impractical to force setuptools to conform to the scheme selected in pip[^1]. So maybe it’s time to finally pull the plug on that method? What remains to do to make that happen? At an absolute minimum, we should make the legacy install method fail if a custom scheme is set by this option.

Testing will be a possible issue. In theory, we should test as much of pip’s functionality as possible with a custom scheme, as well as with the default scheme. But that will be a fairly massive increase in our already large test times[^2]. I feel like it would be worth reorganising the test suite to have a “core functionality” section that’s run on a matrix basis (normal, zipapp, custom scheme, …) with the remaining functional tests being normal scheme only (but if we do that, are we just crossing our fingers that there’s no edge cases?) I don’t feel anything like confident enough in my understanding of our test suite to tackle something like this, though. For now, I’ll probably just add “custom scheme” tests as I go, ignoring the test runtime issue, and leave refactoring the test suite to someone with more expertise…

[^1]: Actually, it may be possible to do this. But it would be complex and difficult to test, and there’s little point in doing this for an approach that we’re actively in the process of removing. [^2]: Particularly if we also add testing the zipapp version, which has the same sort of “everything should work the same” problem.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

OK, I got over-excited by the results of my investigations, and as a result, generalised my findings way beyond what they justified.

@pradyunsg you’re absolutely right, a --python option that simply works out what files to install using the current interpreter and then dumps them in the target interpreter’s environment would be incredibly broken. In my defense, I was thinking of “use the base Python to install stuff in a virtualenv built from it”, which is basically about the only case where that approach stands any chance. So yes, I completely agree, we should be implementing --python by re-invoking pip with the target interpreter.

On the other hand, the Scheme mechanism can be used to create a more fully functional version of --target (and --prefix and --root) which supports the full set of pip commands (pip install --upgrade, pip list --outdated, pip uninstall, pip freeze, etc). That’s what my prototyping demonstrated, and is actually the use case I’m personally more interested in, which is why I was focused on the “set the target paths” mechanism[^1].

Apologies for the confusion. I find issues a fairly clumsy way of handling the sort of “exploring the design space” work that I’m currently doing - but I don’t know of any other, better way of sharing my findings (and unfortunately for the rest of you, I work better by sharing progress as I go along 🙄)

So to summarise:

1. --python should be implemented by re-invoking the current pip with the target interpreter.
2. An improved --target can be implemented by setting the global scheme up front[^2]

Both of these seem like reasonable new features to add - does anyone have any reservations regarding either of them?

[^1]: I still quite like the idea of having a mechanism to set the paths individually, but more for the “pip as a tool run by other tools” use case - if we had a programmatic API, it would be an API, but as we don’t, it’s a CLI option… As it’s for tools to use, it doesn’t need to be particularly user-friendly 🙂 [^2]: Note to self, the legacy setup.py install won’t respect the scheme, so maybe if the global --target is specified we should force PEP 517 mode?

[pradyunsg]

Quick set of thoughts before I move over to my work laptop:

• We don’t need to repackage pip for this at runtime – putting it on the import path at the start and running with that is good-enough. And, we have this now – with the _run-pip script or whatever that’s called.
• I’m strongly opposed to exposing exact scheme paths via the CLI. That’s a discussion we’ve had over on installer (https://github.com/pypa/installer/pull/66, https://github.com/pypa/installer/pull/92, https://github.com/pypa/installer/pull/94). To me, the conclusion was that the potential for pain, caused by someone who doesn’t fully know what they’re doing, is way too big to allow callers to do that. It’s too big of a footgun, that starts by being pointed at your leg.

Beyond that, IIUC, there’s three “points” of coupling with the currently running interpreter:

1. The installation paths, which are also affected by the options passed to pip.
2. The build environment and the Python executable being used in it.
3. The “tags” + markers environment being used to select the distributions in the dependency resolution process.

I think, to make pip work correctly with the various flags that we have, we should consider what our flags do on this front today. I believe we don’t have anything that affects 2. We have flags that should affect 3 and don’t. We also don’t have any way for a user to specify all of these, which we probably should.

A --python would require us to immediately subprocess call with the provided Python interpreter, but it’d give us all of this information “for free”, by doing a short-circuited call to the Python that actually needs to be used. The rest of the code would not need to know that there’s an eager subprocess call happening.