pipenv install fails on windows with a -e git+http repository path

Describe you environment

1. OS Type: Windows 10 x64
2. Python version: Python 3.6.3 (32 bit)
3. Pipenv version: pipenv, version 8.2.7
4. git version: git version 2.14.2.windows.2

Expected result

No exception.

Actual result

> pipenv install --verbose                                                                                              
Pipfile.lock not found, creating…                                                                                       
Locking [dev-packages] dependencies…                                                                                    
Locking [packages] dependencies…                                                                                        
Exception ignored in: <finalize object at 0x4637228; dead>                                                              
Traceback (most recent call last):                                                                                      
  File "d:\python36-32\lib\weakref.py", line 548, in __call__                                                           
    return info.func(*info.args, **(info.kwargs or {}))                                                                 
  File "d:\python36-32\lib\tempfile.py", line 797, in _cleanup                                                          
    _shutil.rmtree(name)                                                                                                
  File "d:\python36-32\lib\shutil.py", line 494, in rmtree                                                              
    return _rmtree_unsafe(path, onerror)                                                                                
  File "d:\python36-32\lib\shutil.py", line 384, in _rmtree_unsafe                                                      
    _rmtree_unsafe(fullname, onerror)                                                                                   
  File "d:\python36-32\lib\shutil.py", line 384, in _rmtree_unsafe                                                      
    _rmtree_unsafe(fullname, onerror)                                                                                   
  File "d:\python36-32\lib\shutil.py", line 384, in _rmtree_unsafe                                                      
    _rmtree_unsafe(fullname, onerror)                                                                                   
  [Previous line repeated 2 more times]                                                                                 
  File "d:\python36-32\lib\shutil.py", line 389, in _rmtree_unsafe                                                      
    onerror(os.unlink, fullname, sys.exc_info())                                                                        
  File "d:\python36-32\lib\shutil.py", line 387, in _rmtree_unsafe                                                      
    os.unlink(fullname)                                                                                                 
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\dildog\\AppData\\Local\\Temp\\tmpebfeqz54source\\hidapi\\.gi
t\\modules\\hidapi\\objects\\pack\\pack-0a555dd332b5157d6b4c17ce683903db1e0ce8bd.idx'                                   
Updated Pipfile.lock (0245b6)!                                                                                          
Installing dependencies from Pipfile.lock (0245b6)…                                                                     
Installing 'click==6.7 --hash=sha256:29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d  --hash=sha256:f15
516df478d5a56180fbf80e68f206010e6d160fc39fa508b65e035fd75130b'                                                          
$ "C:\Users\dildog\.virtualenvs\bcmtool-WeLtC2nH\Scripts\pip.exe" install   --verbose --no-deps  -r C:\Users\dildog\AppD
ata\Local\Temp\pipenv-xq9xnj3q-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w     
Installing 'colorama==0.3.9 --hash=sha256:463f8483208e921368c9f306094eb6f725c6ca42b0f97e313cb5d5512459feda  --hash=sha25
6:48eb22f4f8461b1df5734a074b57042430fb06e1d61bd1e11b078c0fe6d7a1f1'                                                     
$ "C:\Users\dildog\.virtualenvs\bcmtool-WeLtC2nH\Scripts\pip.exe" install   --verbose --no-deps  -r C:\Users\dildog\AppD
ata\Local\Temp\pipenv-4q5i8m0e-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w     
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect                                                    
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect                                               
Requirement already satisfied: click==6.7 in c:\users\dildog\.virtualenvs\bcmtool-weltc2nh\lib\site-packages (from -r C:
\Users\dildog\AppData\Local\Temp\pipenv-xq9xnj3q-requirement.txt (line 1))                                              
  Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatab
le environment, install into an empty virtualenv.                                                                       
Cleaning up...                                                                                                          
                                                                                                                        
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect                                                    
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect                                               
Requirement already satisfied: colorama==0.3.9 in c:\users\dildog\.virtualenvs\bcmtool-weltc2nh\lib\site-packages (from 
-r C:\Users\dildog\AppData\Local\Temp\pipenv-4q5i8m0e-requirement.txt (line 1))                                         
  Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatab
le environment, install into an empty virtualenv.                                                                       
Cleaning up...                                                                                                          
                                                                                                                        
  ================================ 2/2 - 00:00:00                                                                       

Steps to replicate

Create an empty directory, add Pipfile with contents:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]

click = "*"
colorama = "*"
"-e git+http://github.com/crioux/cython-hidapi#egg=hidapi" = "*"

[dev-packages]

[requires]
python_version = "3.6"

and from that folder, just run ‘pipenv install’ It takes a really long time (probably building from git?) and then dies with the exception, having not fully installed the git repo dependency.

[techalchemy]

It isn’t called directly by either actually, as your exception indicates it is called indirectly by tempfile which is supposed to remove this temporary resource which was downloaded (I happen to know that it was downloaded by pip because I have already spent several ~hours~ days working on this specific issue) as part of the dependency resolution process.

I realize that at first glance it may seem that the SO answer provided would be sufficient to fix this issue, but actually that only scratches the surface. In particular, this issue is related to the way subprocess.Popen works on windows. When you call it with shell=True (as we do), it spawns a new process group, which in this case invokes pip to do some stuff. In the case of editable VCS dependencies, pip then does this again, invoking git to grab the repository. On Windows in particular, Popen leaks open file descriptors, which means that the spawned process group of the spawned process group has a leaked open file descriptor to the git index you’re trying to delete.

This doesn’t happen ALL the time, but only sometimes. And when it does, it means that there is no universe in which you can do anything with native error handling to fix this, because the FD is still open. Now you might be thinking, sure, I’ll use psutil, find all the things with open file descriptors on that index, kill those process groups, and then set the permissions, then I can delete them. I have been down that road.

This is all somewhat irrelevant because I have a fix sitting in a PR which is held up currently due to #1372 – I believe this might be what you are interested in

[ed-alertedh]

Well in that case, excellent debugging work and I’m happy to hear there is a fix waiting to be merged! The context manager you’ve implemented looks like a nice solution within the constraints you described - falling back to a warning when an open fd has leaked is definitely much nicer than the current traceback.

Apologies for assuming it was a simple fix, I didn’t realise somebody was working on it as the issue was closed and not referenced anywhere else. I was mainly trying to sound less whiny by offering to help 😃