pip install --upgrade pip breaks pip

This bug looks very close to #5432 “pip install --upgrade pip breaks pip, still”.

Environment

• pip version: pip 19.2.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
• Python version: Python 3.7.4
• OS: MacOS Mojave 10.14.5 (18F132)

I use the Homebrew packet manager: Homebrew 2.1.89-07-23

Description

In the original brew install python installation, /usr/local/bin/pip3 is a symlink. It eventually reaches its target file: /usr/local/Cellar/python/3.7.4/bin/pip3.

The brew info python help recommends to use unversioned symlinks at /usr/local/opt/python/libexec/bin: /usr/local/opt/python/libexec/bin/pip is a symlink to /usr/local/Cellar/python/3.7.4/bin/pip3.

pip install --upgrade pip creates a new file /usr/local/bin/pip instead of of updating the target of the symlink, that is, /usr/local/Cellar/python/3.7.4/bin/pip3. Moreover, this latter file is not updated.

As the PATH finds /usr/local/opt/python/libexec/bin/pip before /usr/local/bin/pip, everything fails…

Expected behavior

On updating pip, /usr/local/bin/pip should be a symlink to /usr/local/opt/python/libexec/bin/pip.

How to Reproduce

1. Clean off all the pip’s find -E /usr/local/ -type f -iregex '.*/pip[23]?' -print -delete

2. Reinstall python with Homebrew brew reinstall python

3. Check the status of /usr/local/bin/pip*, etc. and look for the similar i-nodes. NB: Note there is no /usr/local/bin/pip.

ls -li /usr/local/bin/pip*
ls -li /usr/local/opt/python/libexec/bin/pip
ls -li /usr/local/opt/python/libexec/bin/../../bin/pip3
ls -li /usr/local/bin/../Cellar/python/3.7.4/bin/pip3

1. Upgrade pip with pip /usr/local/opt/python/libexec/bin/pip install --upgrade pip

2. Redo Step 3 above.

3. Try out /usr/local/bin/pip --version

4. Try out /usr/local/opt/python/libexec/bin/pip --version

Output

pommeau:~ bouge$ find -E /usr/local/ -type f -iregex '.*/pip[23]?' -print -delete

/usr/local//bin/pip3
/usr/local//bin/pip
find: /usr/local//etc/FileWaveInstallers/FlatPackage-Microsoft_Office_2019_VL_Serializer.pkg: Permission denied
find: /usr/local//etc/FileWaveInstallers/FlatPackage-Microsoft_AutoUpdate_4.3.18090901_Updater.pkg: Permission denied
find: /usr/local//etc/FileWaveInstallers/FlatPackage-Microsoft_Office_16.17.18090901_Installer.pkg: Permission denied
/usr/local//Cellar/python/3.7.4/bin/pip3

pommeau:~ bouge$ brew reinstall python

==> Reinstalling python 
==> Downloading https://homebrew.bintray.com/bottles/python-3.7.4.mojave.bottle.
Already downloaded: /Users/bouge/Library/Caches/Homebrew/downloads/ffcfe300e2ad205387a2bdd27185d98d64621578d26d2f6ccbbcd091ff799fc8--python-3.7.4.mojave.bottle.tar.gz
==> Pouring python-3.7.4.mojave.bottle.tar.gz
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.4/bin/python3 -s setup.py --no-user-cfg install
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin

If you need Homebrew's Python 2.7 run
  brew install python@2

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python/3.7.4: 3,869 files, 60MB

pommeau:~ bouge$ ls -li /usr/local/bin/pip*

17870036 lrwxr-xr-x  1 bouge  wheel    34B  4 mar 14:11 /usr/local/bin/pip2@ -> ../Cellar/python@2/2.7.16/bin/pip2
17870037 lrwxr-xr-x  1 bouge  wheel    36B  4 mar 14:11 /usr/local/bin/pip2.7@ -> ../Cellar/python@2/2.7.16/bin/pip2.7
36689213 lrwxr-xr-x  1 bouge  wheel    31B 24 jul 15:54 /usr/local/bin/pip3@ -> ../Cellar/python/3.7.4/bin/pip3
36689214 lrwxr-xr-x  1 bouge  wheel    33B 24 jul 15:54 /usr/local/bin/pip3.7@ -> ../Cellar/python/3.7.4/bin/pip3.7
12675428 -rwxr-xr-x@ 1 bouge  wheel   263B 16 jan  2019 /usr/local/bin/pip_upgrade_outdated*

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/pip
36689211 lrwxr-xr-x  1 bouge  staff    14B 24 jul 15:54 /usr/local/opt/python/libexec/bin/pip@ -> ../../bin/pip3

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/../../bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/opt/python/libexec/bin/../../bin/pip3*

pommeau:~ bouge$ ls -li /usr/local/bin/../Cellar/python/3.7.4/bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/bin/../Cellar/python/3.7.4/bin/pip3*

pommeau:~ bouge$ /usr/local/opt/python/libexec/bin/pip install --upgrade pip

Collecting pip
  Using cached https://files.pythonhosted.org/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.1.1
    Uninstalling pip-19.1.1:
      Successfully uninstalled pip-19.1.1
Successfully installed pip-19.2.1

pommeau:~ bouge$ ls -li /usr/local/bin/pip*

36690799 -rwxr-xr-x  1 bouge  wheel   234B 24 jul 15:55 /usr/local/bin/pip*
17870036 lrwxr-xr-x  1 bouge  wheel    34B  4 mar 14:11 /usr/local/bin/pip2@ -> ../Cellar/python@2/2.7.16/bin/pip2
17870037 lrwxr-xr-x  1 bouge  wheel    36B  4 mar 14:11 /usr/local/bin/pip2.7@ -> ../Cellar/python@2/2.7.16/bin/pip2.7
36690800 -rwxr-xr-x  1 bouge  wheel   234B 24 jul 15:55 /usr/local/bin/pip3*
36690801 -rwxr-xr-x  1 bouge  wheel   234B 24 jul 15:55 /usr/local/bin/pip3.7*
12675428 -rwxr-xr-x@ 1 bouge  wheel   263B 16 jan  2019 /usr/local/bin/pip_upgrade_outdated*

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/pip
36689211 lrwxr-xr-x  1 bouge  staff    14B 24 jul 15:54 /usr/local/opt/python/libexec/bin/pip@ -> ../../bin/pip3

pommeau:~ bouge$ ls -li /usr/local/opt/python/libexec/bin/../../bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/opt/python/libexec/bin/../../bin/pip3*

pommeau:~ bouge$ ls -li /usr/local/bin/../Cellar/python/3.7.4/bin/pip3
36689157 -rwxr-xr-x  1 bouge  staff   386B 24 jul 15:54 /usr/local/bin/../Cellar/python/3.7.4/bin/pip3*

pommeau:~ bouge$ /usr/local/bin/pip --version

pip 19.2.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)

pommeau:~ bouge$ /usr/local/opt/python/libexec/bin/pip --version

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pip 19.2.1 (/usr/local/lib/python3.7/site-packages), Requirement.parse('pip==19.1.1'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/python/libexec/bin/pip", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pip==19.1.1' distribution was not found and is required by the application

[bijoustone]

Below steps fixed my problem. Share it out in case anyone needs it:

cp /usr/local/opt/python/libexec/bin/pip /usr/local/opt/python/libexec/bin/pip_ORIG
mv /usr/local/bin/pip /usr/local/opt/python/libexec/bin/pip
ln -s /usr/local/opt/python/libexec/bin/pip /usr/local/bin/pip
pip --version

Credit to this post.

[pradyunsg]

https://github.com/Homebrew/homebrew-core/issues/43867

Fixed downstream.