All virtualenvs break when Python version is upgraded

I ran

$ pipx install poetry

just yesterday, and everything worked fine. Then I ran

$ brew upgrade

which upgraded my system Python from 3.7.2 to 3.7.3. Now pipx no longer works:

% poetry --version
zsh: /Users/raxod502/.local/bin/poetry: bad interpreter: /Users/raxod502/.local/pipx/venvs/poetry/bin/python: no such file or directory

% pipx run poetry --version
⚠️  poetry is already on your PATH and installed at /Users/raxod502/.local/bin/poetry. Downloading and running anyway.
Traceback (most recent call last):
  File "/Users/raxod502/.local/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 525, in cli
    exit(run_pipx_command(parsed_pipx_args, binary_args))
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 141, in run_pipx_command
    use_cache,
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 101, in run
    retval = venv.run_binary(binary, binary_args)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/Venv.py", line 106, in run_binary
    return _run(cmd, check=False)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/Venv.py", line 127, in _run
    returncode = subprocess.run(cmd_str_list).returncode
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/raxod502/.local/pipx/.cache/d7fe01e92227b36/bin/poetry': '/Users/raxod502/.local/pipx/.cache/d7fe01e92227b36/bin/poetry'

% pipx upgrade poetry
Traceback (most recent call last):
  File "/Users/raxod502/.local/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 525, in cli
    exit(run_pipx_command(parsed_pipx_args, binary_args))
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 184, in run_pipx_command
    include_deps=args.include_deps,
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 213, in upgrade
    old_version = venv.get_venv_metadata_for_package(package).package_version
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/Venv.py", line 74, in get_venv_metadata_for_package
    stdout=subprocess.PIPE,
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/raxod502/.local/pipx/venvs/poetry/bin/python': '/Users/raxod502/.local/pipx/venvs/poetry/bin/python'

% pipx list
venvs are in /Users/raxod502/.local/pipx/venvs
binaries are exposed on your $PATH at /Users/raxod502/.local/bin
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 538, in _get_package_summary
    metadata = venv.get_venv_metadata_for_package(package)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/Venv.py", line 74, in get_venv_metadata_for_package
    stdout=subprocess.PIPE,
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/raxod502/.local/pipx/venvs/poetry/bin/python': '/Users/raxod502/.local/pipx/venvs/poetry/bin/python'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/raxod502/.local/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 525, in cli
    exit(run_pipx_command(parsed_pipx_args, binary_args))
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 187, in run_pipx_command
    commands.list_packages(PIPX_LOCAL_VENVS)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 599, in list_packages
    for package_summary in p.map(_get_package_summary, dirs):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/Users/raxod502/.local/pipx/venvs/poetry/bin/python': '/Users/raxod502/.local/pipx/venvs/poetry/bin/python'

% pipx install poetry
'poetry' already seems to be installed. Not modifying existing installation in '/Users/raxod502/.local/pipx/venvs/poetry'. Pass '--force' to force installation

% pipx install poetry --force
Installing to existing directory '/Users/raxod502/.local/pipx/venvs/poetry'
Error: [Errno 2] No such file or directory: '/Users/raxod502/.local/pipx/venvs/poetry/bin/python3.7': '/Users/raxod502/.local/pipx/venvs/poetry/bin/python3.7'
'/usr/local/opt/python/bin/python3.7 -m venv /Users/raxod502/.local/pipx/venvs/poetry' failed

% pipx reinstall-all python3
Traceback (most recent call last):
  File "/Users/raxod502/.local/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 516, in cli
    exit(run_pipx_command(parsed_pipx_args, binary_args))
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 209, in run_pipx_command
    skip=args.skip,
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 467, in reinstall_all
    uninstall(venv_dir, package, local_bin_dir, verbose)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 426, in uninstall
    metadata = venv.get_venv_metadata_for_package(package)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/Venv.py", line 85, in get_venv_metadata_for_package
    stdout=subprocess.PIPE,
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/raxod502/.local/pipx/venvs/poetry/bin/python': '/Users/raxod502/.local/pipx/venvs/poetry/bin/python'

% pipx uninstall-all
Traceback (most recent call last):
  File "/Users/raxod502/.local/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 516, in cli
    exit(run_pipx_command(parsed_pipx_args, binary_args))
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/main.py", line 191, in run_pipx_command
    commands.uninstall_all(PIPX_LOCAL_VENVS, LOCAL_BIN_DIR, verbose)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 449, in uninstall_all
    uninstall(venv_dir, package, local_bin_dir, verbose)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/commands.py", line 426, in uninstall
    metadata = venv.get_venv_metadata_for_package(package)
  File "/Users/raxod502/.local/lib/python/site-packages/pipx/Venv.py", line 85, in get_venv_metadata_for_package
    stdout=subprocess.PIPE,
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/raxod502/.local/pipx/venvs/poetry/bin/python': '/Users/raxod502/.local/pipx/venvs/poetry/bin/python'

I assume the problem is that when pipx creates the virtual environments, it resolves /usr/local/bin/python3 as a symbolic link, which is problematic since the destination of that symlink changes every time Homebrew upgrades Python (thus making the old virtualenv’s Python executable no longer work). I have experienced this problem with Pip as well, but presumably pipx could wrap this behavior somehow to make it work, or at least display a useful error message. At the least, it would be nice to have pipx reinstall-all python3 work properly.

Workaround is something like the following:

% mv ~/.local/pipx /tmp/pipx
% ls /tmp/pipx/venvs | xargs -L 1 pipx install

and manually prune any broken symlinks from ~/.local/bin.

[jaraco]

Personally, I like symlinks and would prefer to keep them. I wonder instead of pipx could have a command to repair venvs after its underlying Python changes. Actually, I guess I’d probably push this upstream and find out what is the best recommendation for any project using venvs to do after a Python upgrade… and then automate that recommendation in pipx.

[itsayellow]

oh boy. a year after this issue is opened I see that there’s been no progress to resolve it without hacky workarounds 😬 in some respects I’m glad because what I thought to be the problem has been confirmed here, but I’m also not gonna worry too much about trying to make this work. I think I don’t use this tool enough to justify the effort to side step the problem every time I do a Python upgrade.

I’m on mac, and homebrew upgrades my python all the time. All my pipx packages are easily fixed when this happens by:

pipx reinstall-all

This workaround doesn’t seem too hacky to me.