Feature proposal: Speed up pipx by using a shared pip for venvs

On Windows, under Python 3.7.3 and later, python -m venv --without-pip is essentially instant (it takes 0.5 sec, where python -m venv takes around 13 sec, on my PC). Saving the cost of installing pip in every venv would therefore be significant (especially for pipx run).

I believe this should be possible, by installing a single copy of pip in ~/.local/pipx, and then putting a .pth file in the newly created environment(s). This may not be 100% compatible with having pip installed in the venv (particularly for packages with dependencies on pip) so it should probably be opt-in, based on a flag somehow. Upgrading the shared pip would need some thought as well, as pip install --target doesn’t support upgrades.

Would this be an acceptable feature to add? I’d be willing to do the implementation work, but I would need to know if it’s something that would be accepted (and I’d likely need guidance on how to design the UI for the feature so that it fits with the rest of pipx).

I’ve no idea if the feature would be useful on non-Windows systems - I don’t have a feel for whether venv creation without pip is fast on other systems (they don’t use the same fast mechanism as Windows does).

[cs01]

This has been added in 0.14.0.0.

[pfmoore]

This sounds pretty promising, although a little magical

It is a bit magical, but .pth files are a fairly standard feature (setuptools uses them for installing projects in “develop” mode, for instance). Basically, all it does is put the directory containing pip onto sys.path, so it shouldn’t cause any fundamental issues.

Can a single pip be reliably used across venvs with different Python versions?

Yes, definitely. Pip has no version-specific code (that’s visibly the case, because it’s distributed as a .py2.py3 wheel) and sharing the code among different versions of Python is OK. The .pyc files will be version-specific, but that should be fine (on Python 3, each version has its own .pyc file in __pycache__, and assuming we’re only supporting Python 2.7, there’s only one version of a Python 2 .pyc file, so no clash there).

pipx upgrades pip every chance it gets. Is is possible for pipx to continue doing this with the new .pth scheme?

It is, and that’s probably easier than (say) making the user manage it themselves. There’s some fiddliness involved, but I’ll sort out the details as I develop the code, and discuss any decisions. It’s definitely the sort of “I’d need advice on design” thing I’d want to discuss before making anything final.

if a user opts-in, does it apply to all pipx-managed packages? or can some packages be installed with pip included, with and some without?

I can see ways of doing either. My inclination would be to assume that the best approach would be to have it apply to every package, but allow the user to say “package X needs its own copy of pip”. That may be over-engineered, though. For 99.9% of packages, they don’t actually want or need pip to be installed - it’s only there so that pipx can manage the code (something like black doesn’t depend on pip at all). Those packages can just use a shared pip with no issues or downsides at all, I suspect. But the occasional package that has pip as an actual dependency (I honestly can’t even think of one right now) might need special handling. It’s something I can think about further, though.

Would batch commands like upgrade-all still work?

I see no reason why they wouldn’t.

I am definitely interested in this.

Excellent! For now, that’s all I really wanted to confirm. I’ll put together a basic proof of concept implementation, and we can discuss the details further once I have some code to point to. It might take me a little while to get something together (my free time is pretty variable right now) but I will post something as soon as I can.

What I really want to avoid is having something fail at run time in a non-obvious way.

Agree 100%. And this is precisely the sort of design principle that I’m interested in, so that I can make sure I stay true to how you want pipx to behave.