EnvironmentError: [WinError 5] Access is denied (for some packages)

Environment

• pip version: 19.2.3
• Python version: CPython 3.6.8
• OS: Windows 10
• Git for Windows version: 2.23.0.windows.1

Python is installed in C:\Tools\Python\3.6, and is writable to me as a user. Global packages are often installed there and not with --user. Git for Windows is installed in C:\Users\davisda4\Tools\Git, and many essential tools from Linux will be found there. See the output section for a rendering of directories in the path.

Description

pip install fails to unpack some packages with the typical environment error in the title, but most packages work. Sometimes, rebooting will resolve the issue. The package Django is often affected.

Expected behavior

pip install should work reliably on Windows.

Troubleshooting

Often, I just reboot the system, and it works. This time, it was very persisted so I attempted to analyze and troubleshoot extensively.

• Attempted removing files and directories from %TMP% to clean-up space
• Attempted redefining %TEMP% and %TMP% to be someplace easy to access
• Attempted bypassing cache using --no-cache-dir
• Attempted rebooting the system (finally getting frustrated)
• Attempted upgrading global packages pip, setuptools, wheel, virtualenvwrapper-win, virtualenv, virtualenv-clone
• Attempted running outside of Cmder.net in a pure CMD prompt
• Checked access to %TEMP% directory and available space on the filesystem, reported information is below.

How to Reproduce

The problem is intermittent but has occurred on multiple machines, all running Windows 10. All systems I have tried are within the NLM desktop computing environment, and so there may be security check programs stopping access. This almost always affects Django rather than other packages in my experience.

I believe this to be the symptom of a real bug (whatever the source) that is often masked by the usual permission and space problems. Other users will run into this problem, both in my corporate environment and others. I open this in the spirit of trying to get to the root of the issue.

Output

Output below shows the installation of flake8 working and then Django failing. This is outside the virtualenv, but the same thing happens within a virtualenv, and also with an alternate temporary directory.

C:\Users\davisda4>pip --version
pip 19.2.3 from c:\tools\python\3.6\lib\site-packages\pip (python 3.6)

C:\Users\davisda4>pip install flake8
Looking in indexes: https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/simple
Collecting flake8
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/26/de/3f815a99d86eb10464ea7bd6059c0172c7ca97d4bdcfca41051b388a653b/flake8-3.7.8-py2.py3-none-any.whl (70kB)
     |████████████████████████████████| 71kB 770kB/s
Collecting pyflakes<2.2.0,>=2.1.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/84/f2/ed0ffb887f8138a8fe5a621b8c0bb9598bfb3989e029f6c6a85ee66628ee/pyflakes-2.1.1-py2.py3-none-any.whl (59kB)
     |████████████████████████████████| 61kB 787kB/s
Collecting mccabe<0.7.0,>=0.6.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting entrypoints<0.4.0,>=0.3.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Collecting pycodestyle<2.6.0,>=2.5.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/pycodestyle-2.5.0-py2.py3-none-any.whl (51kB)
     |████████████████████████████████| 51kB 825kB/s
Installing collected packages: pyflakes, mccabe, entrypoints, pycodestyle, flake8
Successfully installed entrypoints-0.3 flake8-3.7.8 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1

C:\Users\davisda4>pip install Django
Looking in indexes: https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/simple
Collecting Django
  Using cached https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/94/9f/a56f7893b1280e5019482260e246ab944d54a9a633a01ed04683d9ce5078/Django-2.2.5-py3-none-any.whl
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\davisda4\\AppData\\Local\\Temp\\pip-unpack-cif5c4t6\\Django-2.2.5-py3-none-any.whl'
Consider using the `--user` option or check the permissions.

Troubleshooting step of checking space and permissions for the pip unpack directory (note that pip was able to create the directory):

C:\Users\davisda4\AppData\Local\Temp\pip-unpack-cif5c4t6>ls

C:\Users\davisda4\AppData\Local\Temp\pip-unpack-cif5c4t6>icacls .
. NT AUTHORITY\SYSTEM:(OI)(CI)(F)
  BUILTIN\Administrators:(OI)(CI)(F)
  NIH\davisda4:(OI)(CI)(F)

Successfully processed 1 files; Failed processing 0 files

C:\Users\davisda4\AppData\Local\Temp\pip-unpack-cif5c4t6>df -kh .
Filesystem                            Size  Used Avail Use% Mounted on
C:/Users/davisda4/AppData/Local/Temp  477G   92G  385G  20% /tmp

Please note that “NIH\davisda4:(OI)(CI)(F)” means that I am the “inherited” owner (OI) and creator (CI) of the directory, and have full access (F).

The system path:

C:\Program Files\VanDyke Software\Clients\
C:\tools\Node\8.15\
C:\Tools\Python\3.6\
C:\tools\Python\3.6\Scripts\
C:\tools\Python\3.5\
C:\tools\Python\3.5\Scripts\
C:\tools\Perl\5.2\Perl64\bin\
C:\tools\Ruby\2.4\bin\
C:\tools\Ruby\2.4\bin
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files (x86)\HID Global\ActivClient\
C:\Program Files\HID Global\ActivClient\
C:\Users\nlmoccsswdist.nih\AppData\Local\Microsoft\WindowsApps
C:\Users\nlmoccsswdist.nih\AppData\Roaming\npm
C:\Program Files\dotnet\
C:\Program Files\Microsoft SQL Server\130\Tools\Binn\
C:\Program Files (x86)\Webex\Webex\Applications
c:\Users\nlmoccsswdist.nih\AppData\Roaming\npm
C:\Tools\Java\11\bin\
C:\Tools\Java\11\bin
C:\Tools\Java\11\jre\bin

The user path:

C:\Tools\Python\3.6
C:\Tools\Python\3.6\Scripts
C:\Tools\Perl\5.2\Perl64\bin
C:\Users\davisda4\Tools\apache-ant-1.10.5\bin
C:\Users\davisda4\Tools\apache-maven-3.5.4\bin
C:\Users\davisda4\Tools\SeleniumDrivers
C:\Tools\instantclient_12_2
C:\Users\davisda4\AppData\Roaming\npm
C:\Program Files\Sublime Text 3
C:\Users\davisda4\Tools\xmlsoft\bin
C:\Users\davisda4\Tools\HashiCorp\Vagrant\bin
C:\Users\davisda4\Tools\apache-jena-3.8.0\bat
C:\Users\davisda4\Tools\bin
C:\Users\davisda4\Tools\Git\cmd
C:\Users\davisda4\Tools\Git\mingw64\bin
C:\Users\davisda4\Tools\Git\usr\bin

[danizen]

We now suspect that Symantec Endpoint Protection is responsible for the problem, but we have no proof, just circumstantial evidence. My security guys are planning to remove it in favor of Cylance next week, so I am going to hang tight and try again then. Beyond that, we’ve discovered that this does affect other systems on our network, and is not just me. I’d like to hold this open at least until we can see if switching from Symantec Endpoint Protection to Cylance addresses the issue, and then see about creating a troubleshooting guide for this issue.

[choyrim]

I know this issue is closed. But these symptoms are similar to an issue I’m also observing and this is the only useful thread to post to.

The problem can be reproduced reasonably reliably on windows (with CylanceProtect) by trying to install something with a big download. it depends on how slow CylanceSvc.exe is running. most of the time i can reproduced it with just pip download. If I lower the priority of CylanceSvc I can reproduce close to a 100% of the time.

> set TEMP=c:\tmp\pip
> pip download awscli
Collecting awscli
  Using cached https://files.pythonhosted.org/packages/f2/f5/d682aa32100edc6908a784630aa2903ef7d2735130e6df98a05af6c33096/awscli-1.16.271-py2.py3-none-any.whl
  Saved c:\users\rich9002\tmp\awscli-1.16.271-py2.py3-none-any.whl
ERROR: Exception:
Traceback (most recent call last):
  File "c:\sw\python37\lib\shutil.py", line 398, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-4gdrskd3\\awscli-1.16.271-py2.py3-none-any.whl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\sw\python37\lib\site-packages\pip\_internal\cli\base_command.py", line 188, in main
    status = self.run(options, args)
  File "c:\sw\python37\lib\site-packages\pip\_internal\commands\download.py", line 156, in run
    resolver.resolve(requirement_set)
  File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "c:\sw\python37\lib\site-packages\pip\_internal\operations\prepare.py", line 199, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "c:\sw\python37\lib\site-packages\pip\_internal\download.py", line 1064, in unpack_url
    progress_bar=progress_bar
  File "c:\sw\python37\lib\site-packages\pip\_internal\download.py", line 935, in unpack_http_url
    os.unlink(from_path)
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\temp_dir.py", line 60, in __exit__
    self.cleanup()
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\temp_dir.py", line 83, in cleanup
    rmtree(self.path)
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "c:\sw\python37\lib\site-packages\pip\_vendor\six.py", line 693, in reraise
    raise value
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\misc.py", line 166, in rmtree
    onerror=rmtree_errorhandler)
  File "c:\sw\python37\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\sw\python37\lib\shutil.py", line 400, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\misc.py", line 176, in rmtree_errorhandler
    os.chmod(path, stat.S_IWRITE)
PermissionError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-4gdrskd3\\awscli-1.16.271-py2.py3-none-any.whl'
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

using Process Monitor (procmon), I confirmed that CylanceSvc.exe (virus protection) was still reading the file as python kept trying to delete the files.

Two workarounds (1) hack the code, or (2) install the big whl file directly. The first work around is to download the .whl file (with curl or a failed pip download) and install the .whl file directly. The second workaround is the change the retry time in the pip\_internal\utils\misc.py file.

# Retry every half second for up to 3 seconds
#@retry(stop_max_delay=3000, wait_fixed=500) # original
@retry(stop_max_delay=60000, wait_fixed=500) # wait up to a minute
def rmtree(dir, ignore_errors=False):
    # type: (str, bool) -> None
    shutil.rmtree(dir, ignore_errors=ignore_errors,
                  onerror=rmtree_errorhandler)