pip 19.x --upgrade breaks on nfs

Environment

• pip version: 19.x
• Python version: 2.7, 3.6
• OS: Linux

Description pip19 uses AdjacentTempDirectory, where previous pip versions used tempfile. Because of nfs “silly rename”, if working on an nfs mount and pip is called via a subprocess from a python script that happens to use one of the packages being upgraded, pip fails when trying to cleanup (since the C extensions are still mapped to interpreter – can be seen in /proc/$PID/maps)

Expected behavior

Expect this either to not fail (as in previous versions of pip) or just raise more appropriate exception. Right now, this causes pip to enter this error handler (that seems to be designed for windows). chmod’ing the path to 200 works for Windows, but is not enough to delete on Linux. The chmod done here ultimately causes pip to raise Permission Error. Maybe that should be chmod to stat.S_IRWXU?

How to Reproduce

in some nfs location, install simplejson (or anything that uses a C extension)

python3.6 -m virtualenv env
source env/bin/activate
pip install simplejson==3.16.0

then import it, and call pip in a subprocess

import subprocess
import sys

import simplejson  # <-- causes this to break


INSTALL_COMMAND = [
    sys.executable,
    "-m",
    "pip",
    "install",
    "--disable-pip-version-check",
    "--upgrade",
    "--force-reinstall",
    "simplejson==3.16.0",
]
print("return code:", subprocess.Popen(INSTALL_COMMAND).wait())

You can also run the script interactively and see the mapped files. I saved above code as ‘mini.py’

$ python -i mini.py &
# wait for the pip process to finish
$ cat /proc/$!/maps | grep nfs
# will show the .nfs* files

Output

Looking in indexes: https://...
Collecting simplejson==3.16.0
Installing collected packages: simplejson
  Found existing installation: simplejson 3.16.0
    Uninstalling simplejson-3.16.0:
      Successfully uninstalled simplejson-3.16.0
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'env/lib/python3.6/site-packages/~implejson'
Consider using the `--user` option or check the permissions.

return code: 1

Taking a look at that location:

$ ls env/lib/python3.6/site-packages/~implejson
ls: cannot open directory 'env/lib/python3.6/site-packages/~implejson': Permission denied
$ chmod 700 env/lib/python3.6/site-packages/~implejson
$ ls -a env/lib/python3.6/site-packages/~implejson
.  ..  .nfsxxxxxxxxxxxxxxxxxxxxxxxx  # <-- some nfs silly-renamed file

Paste the output of the steps above, including the commands themselves and
pip's output/traceback etc.

[kottmast]

I think I’m hitting this issue, but I’m confused about the affected version and/or I’m hitting something related. I’m on an ubuntu machine, with nfs mounted /home and trying to create a conda environment from a environment.yml file. The conda part of the installation works, but pip fails on version 19.3.1 with the .nfsxxx device or resource is busy error. I tried downgrading to several previous versions of pip, but it then produces the “directory is not empty” error. Has this ever worked? Is there some workaround available?

[cjerdonek]

This looks like the same as issue #6321 which was recently filed, but this issue has more information. Thanks a lot for the detailed report and analysis, @ntextreme3!