pip install --user tries to uninstall from system site-packages

• Pip version: 9.0.1
• Python version: 3.5
• Operating system: Mac (Macports)

Description:

I’ve been a happy user of pip for a while, thank you for all your work!

Now (possibly after a recent pip update), the bahaviour has changed and

pip install . --user

from a source folder in a Python project I work on (Gammapy) tries to uninstall a stable 0.5 version of Gammapy I have in the Macports site-packages.

I wasn’t expecting pip to do this uninstall and it’s not clear to me from the pip 9.0 and 9.0.1 release notes that this is an intended change in behaviour.

Is this a bug or am I doing it wrong? What’s the workaround?

What I’ve run:

$ pip --version
pip 9.0.1 from /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (python 3.5)
$ pip install . --user
Processing /Users/deil/code/gammapy
Requirement already satisfied: numpy>=1.8 in /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from gammapy==0.6.dev4330)
Requirement already satisfied: astropy>=1.3 in /Users/deil/Library/Python/3.5/lib/python/site-packages (from gammapy==0.6.dev4330)
Requirement already satisfied: regions in /Users/deil/Library/Python/3.5/lib/python/site-packages (from gammapy==0.6.dev4330)
Requirement already satisfied: click in /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from gammapy==0.6.dev4330)
Installing collected packages: gammapy
  Found existing installation: gammapy 0.5
    Uninstalling gammapy-0.5:
Exception:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 544, in move
    os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/gammapy' -> '/var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-lc_5uj1j-uninstall/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/gammapy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_install.py", line 754, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 556, in move
    rmtree(src)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 438, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: '__init__.py'

[cdeil]

Ha ha, I haven’t used --user since 2 years, switched to use conda envs and conda and pip install things there and that works fine. So personally I don’t really care about --user any more.

But re-reading the discussion above: yeah, I would still say a PermissionError from system site packages when trying to pip install --user something is a bug. Similar to what others said above, pip install --user should just install the package and not look at all what’s in system site-packages. I think the initial counter-argument that foo.bar gets mixes is not true, Python doesn’t do that, it’s OK to have the same package installed multiple times in different envs in the sys.path list, like in this case system site-packages and now user site-packages.

@cdeil would it be an improvement to have pip send a message on stderr that tells the user that it is deleting the older version?

Well, yes, if that is an easy addition, and the real fix and change in behaviour of pip install --user to not delete isn’t coming within the near future (say ~ 1 year), then you could add that message now.

[pradyunsg]

The --ignore-installed flag isn’t a exact fix for the problem here.

#4575 and #1668 are the proper fixes for this and will need someone to either do it or fund work to get it done. (https://wiki.python.org/psf/Fundable Packaging Improvements)