Once system python and pipx upgraded, cannot install new app if some previous old shared_venv existed

Describe the bug

On Manjaro/Arch, I was with python 3.7.4. To ensure stability of some apps (and as some apps are not fully working on 3.8 that I new was coming) I also had python 3.7.5 installed through pyenv. I use this python to install my apps with

pipx install --python ~/.pyenv/versions/3.7.5/bin/python flexget
pipx inject flexget transmissionrpc

Then my system got upgraded from 3.7.4 to 3.8 (Note that my 3.7.5 pyenv stayed) I reinstalled pipx on 3.8 on system. Then I had error thrown when installing new pipx no matter what versions I tried:

$ pipx install pycowsay --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /usr/bin/python -m venv --without-pip /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay
/home/hcooh/.local/pipx/venvs/pycowsay/bin/python: No module named pip

pipx > (rmdir:16): removing directory /home/hcooh/.local/pipx/venvs/pycowsay
'/home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay' failed

or even trying to use the pyenv:

pipx install --python ~/.pyenv/versions/3.7.5/bin/python pycowsay --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.pyenv/versions/3.7.5/bin/python -m venv --without-pip /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay
/home/hcooh/.local/pipx/venvs/pycowsay/bin/python: No module named pip

pipx > (rmdir:16): removing directory /home/hcooh/.local/pipx/venvs/pycowsay
'/home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay' failed

Running the already installed app (using the 3.7.5) would work fine and even pipx runpip xxx list I realised then that .local/pipx/shared/pyvenv.cfg mentioned python version 3.7.4. So I renamed the .local/pipx/shared folder differently. It recreated the shared lib for python3.8. I had to then symlink the pyenv folder of python3.7 with:

ln -s ../../../../.pyenv/versions/3.7.5/lib/python3.7/ ~/.local/pipx/shared/lib/python3.7

And then install run fine and the already installed app run fine too

How to reproduce

See above

Expected behavior

pipx Install install should work after a pyhton upgrade even if hte previous shared folder still exists. i think

[nyanpasu64]

Summary

Judging from https://github.com/pypa/pipx/issues/294#issuecomment-992699160 and https://github.com/pypa/pipx/issues/294#issuecomment-597792241, there’s at least 3 versions in a pipx setup:

• The Python interpreter version used to install pipx
• The shared env’s version
• Each app’s venv version

Each of these should match the system Python interpreter, but might desync (the system Python interpreter’s path is still present, but belongs to a different Python version) when updating Python in-place on Linux.

Ideas

Not a pipx contributor so far, but I could try changing reinstall-all to always recreate the shared environment, and change more pipx operations (reinstall, upgrade[-all], install, etc.) to automatically recreate the shared environment if the Python interpreter version has changed. (Perhaps it should also reinstall all packages? It’s necessary to fix broken packages, but unexpected by the user. Maybe do nothing in shell scripts and prompt the user if a TTY is detected?) You’ll either have to save the install-time Python version in a config file, or detect the install-time Python version from the shared/app venv (I’m not sure how, and looking for a .../lib/python3.x folder can mistakenly indicate a venv was installed on newer Python than it actually was; on current pipx I managed to end up with a shared env with an empty python3.10 folder, and an app’s venv with a python3.10 folder containing Python 3.10 __pycache__ but 3.9 C extensions).

My other idea is to add a config file/database containing a list of venvs and each venv’s install options and Python versions. This complicates the architecture since the config file can desync with the actual folders present (eg. if a user adds/removes venv folders by hand), but is useful since if a failed reinstall deletes a venv, it’s still present in the config/database and a subsequent reinstall-all knows to install them. This config file should also store the shared environment’s Python version and/or interpreter path, so pipx can compute when to regenerate the shared env and all app venvs (or you can skip this and use the version stored in ~/.local/pipx/shared/pyvenv.cfg).

In theory you could also store the Python version/path of each app’s venv (though this is already tracked in ~/.local/pipx/venvs/*/pyvenv.cfg. Using this to support different venvs with different Python versions/interpreters is too complex IMO. It might be useful to use this to track which venvs need to be reinstalled. I kinda prefer not supporting partially upgrading venvs, but instead block all pipx operations until the shared env and all venvs are upgraded in a single operation. But old pipx versions (combined with users copying commands to try to fix pipx) can generate mixed Python versions. If we don’t check per-venv Python versions, we can update the docs and tell users to “reinstall-all to fix things”; if we do, we can detect and address this situation.

I think it’s still a good idea for reinstall-all to regenerate the shared env, even if it’s supposedly up-to-date according to the “install-time Python version” metadata. This allows the command to fix errors tied to the shared env, rather than any app’s venv. (These errors may exist, I’ve never encountered any other than a Python version mismatch.) This also fixes cases where the Python version has actually changed even though the metadata wrongly indicates the install-time version matches the current version.

Is it possible for each app’s shim binary to automatically reinstall the app (and shared env, maybe other apps) if necessary? This makes the reinstallation process transparent to users and hopefully “just work”, but I’m not sure if users want their apps to magically change behavior in edge cases (though they already break right now). And this might slow-down happy-path app startup time (though you can get pretty close to zero overhead by only reinstalling when you catch ModuleNotFoundError) (though I don’t know if this fixes both “outdated shared env and app” and “updated shared env but outdated app”). And this doesn’t catch cases where pipx install --system-site-packages and pipx upgrade results in a working app which uses C libraries from the system rather than the venv.

Or perhaps pipx itself should catch ModuleNotFoundError: No module named 'pipx' and reinstall the shared env and all packages. (I don’t think I’ve seen this particular error since I installed pipx from the Arch repositories rather than pip install --user.) This fixes another user-facing failure mode of pipx, but can’t replace my previous suggestions since it fails to catch cases where pipx is updated but the shared env is out-of-date and unusable.

I didn’t look into handling the system Python interpreter being uninstalled, and a new one being installed to a different location. This case is more common on Windows where Python is installed to C:\Python3xx\bin\python rather than /usr/bin/python. This may break the pipx binary, and require the user to reinstall pipx by hand, and then ideally pipx updates the shared and app envs from there.

Testing

I think it will be difficult to come to a consensus on an optimal design (probably manual testing will help in designing an intuitive interface), and difficult to manually or automatically test updating Python to a new major version (which only happens once per year per Linux installation). Perhaps pyenv will help to test switching Python interpreters quickly. There’s also “build a Docker image with an outdated Python”, but I’m less familiar with that than pyenv, and setting up Docker/Podman requires more upfront setup and invasive system changes (cgroups etc.) than pyenv IMO.

[NotAFile]

Just had this issue again upgrading to fedora 35:

/home/nota/.local/pipx/shared/bin/python: No module named pip
Failed to upgrade shared libraries
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pipx/shared_libs.py", line 113, in upgrade
    subprocess_post_check(upgrade_process)
  File "/usr/lib/python3.10/site-packages/pipx/util.py", line 203, in subprocess_post_check
    raise PipxError(
pipx.util.PipxError: '/home/nota/.local/pipx/shared/bin/python -m pip --disable-pip-version-check install -q --upgrade pip setuptools wheel' failed
/home/nota/.local/pipx/venvs/pyupgrade/bin/python: No module named pip
Error encountered when upgrading pyupgrade:
'/home/nota/.local/pipx/venvs/pyupgrade/bin/python -m pip install --upgrade pyupgrade -q' failed

I fixed it by running:

rm -r ~/.local/pipx/shared/

and then running a pipx operation to make it recreate the shared env with the new python version.