[WinError 5] Access is denied during cleanup/rmtree

Environment

• pip version: 19.2.3+
• Python version: python 3.7
• OS: Windows 10
• AntiVirus: CylancePROTECT

Description pip install or download fails on Windows during cleanup because (real-time) virus protection is still scanning files that pip is trying to remove.

using Process Monitor (procmon), I was able to confirm that CylanceSvc.exe (virus scanner) was reading the files while python was trying to the same files.

I tried both increasing the retry wait (in misc.py) and ignoring the rmtree error (in temp_dir.py). both approaches worked.

This issue indicates that other anti-virus scanners cause similar problems: https://github.com/aws/aws-cli/issues/2654

Expected behavior The package should install.

How to Reproduce pip install or download a large package. I was usually able to reproduce when installing or downloading awscli. If I have difficulty reproducing, I can lower the priority of the Cylance.exe to Background (4), and the symptoms are easier to reproduce consistently.

using install: pip install awscli

using download: pip download awscli

pip download is better because it provides a stacktrace.

Output

using pip install, the error message is quite terse.

>pip install awscli
Collecting awscli
  Using cached https://files.pythonhosted.org/packages/f2/f5/d682aa32100edc6908a784630aa2903ef7d2735130e6df98a05af6c33096/awscli-1.16.271-py2.py3-none-any.whl
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-5e4a1ye3\\awscli-1.16.271-py2.py3-none-any.whl'
Consider using the `--user` option or check the permissions.
. . .

using pip download, the stacktrace is more helpful.

>pip download awscli
Collecting awscli
  Using cached https://files.pythonhosted.org/packages/f2/f5/d682aa32100edc6908a784630aa2903ef7d2735130e6df98a05af6c33096/awscli-1.16.271-py2.py3-none-any.whl
  Saved c:\users\rich9002\tmp\awscli-1.16.271-py2.py3-none-any.whl
ERROR: Exception:
Traceback (most recent call last):
  File "c:\sw\python37\lib\shutil.py", line 398, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-4gdrskd3\\awscli-1.16.271-py2.py3-none-any.whl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\sw\python37\lib\site-packages\pip\_internal\cli\base_command.py", line 188, in main
    status = self.run(options, args)
  File "c:\sw\python37\lib\site-packages\pip\_internal\commands\download.py", line 156, in run
    resolver.resolve(requirement_set)
  File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 196, in resolve
    self._resolve_one(requirement_set, req)
  File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 359, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 307, in _get_abstract_dist_for
    self.require_hashes
  File "c:\sw\python37\lib\site-packages\pip\_internal\operations\prepare.py", line 199, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "c:\sw\python37\lib\site-packages\pip\_internal\download.py", line 1064, in unpack_url
    progress_bar=progress_bar
  File "c:\sw\python37\lib\site-packages\pip\_internal\download.py", line 935, in unpack_http_url
    os.unlink(from_path)
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\temp_dir.py", line 60, in __exit__
    self.cleanup()
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\temp_dir.py", line 83, in cleanup
    rmtree(self.path)
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "c:\sw\python37\lib\site-packages\pip\_vendor\six.py", line 693, in reraise
    raise value
  File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\misc.py", line 166, in rmtree
    onerror=rmtree_errorhandler)
  File "c:\sw\python37\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\sw\python37\lib\shutil.py", line 400, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\sw\python37\lib\site-packages\pip\_internal\utils\misc.py", line 176, in rmtree_errorhandler
    os.chmod(path, stat.S_IWRITE)
PermissionError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-4gdrskd3\\awscli-1.16.271-py2.py3-none-any.whl'

[pradyunsg]

it would be awfully easy for a message like that to get seen as “pip is advising people to turn off their virus scanners” 🙁

That’s True. OTOH, I think we can phrase this nicely.

if WINDOWS:
    print("We've received reports that aggressive antivirus software can also result in such errors, in which case, re-running the pip command usually succeeds.")

😉

[danizen]

I’ve written a utility that can create similar problems even without Cylance Protect, and can make the failure more consistent. This is https://github.com/danizen/watchandlock.

I can run it with my own package, confsecrets, which is quite small. In one command-prompt in a new virtual environment:

pip install requirements.txt
python watchandlock.py --delay 10.0 %TEMP%

In another command-prompt in the same virtual environment

pip install confsecrets

TL;DR - I’m not getting exactly the same error, possibly because I’m opening the file and not the directory. I’ve also confirmed using procmon that Cylance Protect is simply doing the following:

1. Open the file (https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea)

    Operation:        CreateFile
    Desired Access:   Generic Read
    Disposition:      Open
    Options:          Synchronous IO Non-Alert, Non-Directory File, Open No Recall
    Attributes:       n/a
    ShareMode:        Read, Write, Delete
    AllocationSize:   n/a
    OpenResult:       Opened
   

2. Read the file (4 Kb at a time) (https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-readfile)

    Operation:        ReadFile
    Duration:         0.0000083
    Offset:           0
    Length:           4,096
    Priority:         Normal