Pipenv reinstalling all dependencies from Pipfile when installing a new package causes issues on Windows

Issue description

When installing a new package via Pipenv, after installing that new package Pipenv somehow reinstalls all packages mentioned in the Pipfile (as mentioned by the line "Installing dependencies from Pipfile), even those unrelated to that new package. This may work fine on Unix-based OSes, but on Windows there is that problem that Windows locks access to files that are currently in use by another process, so for instance when trying to install a new package while running a Jupyter notebook, the process will end up failing because the reinstalling step tried to overwrite a locked file, resulting in a “[WinError 5] Access is denied” error.

Note that the new package is usually installed correctly, and normally nothing gets actually broken, but it is annoying, especially since there is no reason why installing a new package should force a reinstall of all packages in the virtual environment, especially the unrelated ones. It makes the process much slower than it could be, and it causes the kind of issues I described above.

Expected result

I have a virtual environment containing a few packages including PyMuPDF (an independent PDF-handling library with no dependencies and no dependents), and a Jupyter notebook running that makes use of PyMuPDF. I want to test another PDF library, PDFQuery, which is independent from PyMuPDF. If I do pipenv install pdfquery, my expectation is for Pipenv to install it, its dependencies if they are not already present, and I should get a message that the installation has succeeded.

Actual result

Instead, here is the result:

$ pipenv install pdfquery
Installing pdfquery�
Adding pdfquery to Pipfile's [packages]�
Installation Succeeded
Installing dependencies from Pipfile�
An error occurred while installing pymupdf! Will try again.
An error occurred while installing C:/Users/Christophe.Grandsire/Downloads/numpy-1.15.4-cp37-cp37m-win_amd64.whl! Will try again.
Installing initially failed dependencies�
[pipenv.exceptions.InstallError]:   File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\cli\command.py", line 254, in install
[pipenv.exceptions.InstallError]:       editable_packages=state.installstate.editables,
[pipenv.exceptions.InstallError]:   File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 862, in do_install_dependencies
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, False, failed_deps_queue, retry=False)
[pipenv.exceptions.InstallError]:   File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting pymupdf', '  Using cached https://files.pythonhosted.org/packages/f0/45/b5a90538a9603708dfc705d7d318f4192a5ffcd4c41b360b80624040fe6c/PyMuPDF-1.14.4-cp37-cp37m-win_amd64.whl', 'Installing collected packages: pymupdf', '  Found existing installation: PyMuPDF 1.14.3', '    Uninstalling PyMuPDF-1.14.3:', '
  Successfully uninstalled PyMuPDF-1.14.3']
[pipenv.exceptions.InstallError]: ["Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\\\Users\\\\CHRIST~1.GRA\\\\AppData\\\\Local\\\\Temp\\\\pip-uninstall-adfcfhxg\\\\users\\\\christophe.grandsire\\\\.virtualenvs\\\\equipment_mapping-lnlbcdyh\\\\lib\\\\site-packages\\\\fitz\\\\_fitz.pyd'", 'Consider using the `--user` option or check the permissions.']
ERROR: ERROR: Package installation failed...

As you can see, installing PDFQuery prompted a reinstall of PyMuPDF (despite the two packages being completely unrelated. They don’t even have common dependencies), but since PyMuPDF is currently in use in my open Jupyter notebook, the _fitz.pyd file (a part of PyMuPDF) is locked by Windows, and the attempt to overwrite it causes the [pipenv.exceptions.InstallError]: ["Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\\\Users\\\\CHRIST~1.GRA\\\\AppData\\\\Local\\\\Temp\\\\pip-uninstall-adfcfhxg\\\\users\\\\christophe.grandsire\\\\.virtualenvs\\\\equipment_mapping-lnlbcdyh\\\\lib\\\\site-packages\\\\fitz\\\\_fitz.pyd'", 'Consider using the –user option or check the permissions.'], and the following ERROR: ERROR: Package installation failed..., which is particularly egregious since PDFQuery was actually installed correctly and works fine.

Steps to replicate

I can’t provide a full list of steps to replicate as it would require me to provide the Jupyter notebook I’m using and the data it relies on, and I’m not allowed to share them outside my company. But it should be possible to reproduce this error on any Windows computer (I’m on Windows 10) with a virtual environment containing a specific package currently in use by a Jupyter notebook, and then try to install any other package in that environment. You may need to make sure that specific package contains a .pyd file, which is basically a Windows DLL and is probably the reason why Windows refuses to overwrite it.

---

Here is the output of pipenv --support:

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'C:\\Users\\Christophe.Grandsire\\AppData\\Roaming\\Python\\Python37\\site-packages\\pipenv'

Python location: 'c:\\program files\\python37\\python.exe'

Python installations found:

• 3.7.0: C:\Program Files\Python37\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.16299',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'win32'}

System environment variables:

• ABAQUSLM_LICENSE_FILE
• ACLOCAL_PATH
• ADSKFLEX_LICENSE_FILE
• AD_PATH
• ALLUSERSPROFILE
• ALT_LICENSE_FILE
• ANSICON
• ANSICON_DEF
• APPDATA
• BEICIP_LICENSE_FILE
• BLM_LICENSE_FILE
• BLUECIEL_LICENSE_FILE
• BMGEO_LICENSE_FILE
• CAESAR_LICENSE_FILE
• CALLMD_LICENSE_FILE
• CDLMD_LICENSE_FILE
• CGLD_LICENSE_FILE
• CMG_LIC_HOST
• COMMONPROGRAMFILES
• COMPUTERNAME
• COMSPEC
• CONFIG_SITE
• CUDA_PATH
• CUDA_PATH_V9_2
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• CONEMUANSI
• CONEMUANSILOG
• CONEMUARGS2
• CONEMUARGS
• CONEMUBACKHWND
• CONEMUBASEDIR
• CONEMUBASEDIRSHORT
• CONEMUBUILD
• CONEMUCFGDIR
• CONEMUCONFIG
• CONEMUDIR
• CONEMUDRAWHWND
• CONEMUDRIVE
• CONEMUHWND
• CONEMUHOOKS
• CONEMUPID
• CONEMUPALETTE
• CONEMUSERVERPID
• CONEMUTASK
• CONEMUWORKDIR
• CONEMUWORKDRIVE
• DHSDELFT_LICENSE_FILE
• DISPLAY
• DNVSLM_LICENSE_FILE
• ELIIS_LICENSE_FILE
• ENRESINT_LICENSE_FILE
• EXEPATH
• FLEXLM_NO_CKOUT_INSTALL_LIC
• GDAL_DRIVER_PATH
• GEOCOSM_LICENSE_FILE
• GEOMATICLM_LICENSE_FILE
• GEOSOFT_LICENSE_FILE
• GEOTOMO_LICENSE_FILE
• HAMP_RUSS_LICENSE_FILE
• HOME
• HOMEDRIVE
• HOMEPATH
• HOMESHARE
• HOSTNAME
• HPQ_LICENSE_FILE
• HRH_LICENSE_FILE
• IDL_LMGRD_LICENSE_FILE
• IHS_LICENSE_FILE
• IKON_LICENSE_FILE
• IMSTATUS
• INFOPATH
• INTEL_LICENSE_FILE
• IPASSI_LICENSE_FILE
• IVS_LICENSE_FILE
• JASONLM_LICENSE_FILE
• JAVA_HOME
• JOA_LICENSE_FILE
• JUSTINT_LICENSE_FILE
• KAPPA_LICENSE_FILE
• KNOWSYS_LICENSE_FILE
• LANG
• LCT_LMD_LICENSE_FILE
• LGCX_LICENSE_FILE
• LICSRV_LICENSE_FILE
• LMGRD.SLB_LICENSE_FILE
• LM_PROJECT
• LOCALAPPDATA
• LOGONSERVER
• LSHOST
• MANPATH
• MARINLMD_LICENSE_FILE
• MCSLMD_LICENSE_FILE
• MDAXBBWC_LICENSE_FILE
• MFRACLM_LICENSE_FILE
• MINGW_CHOST
• MINGW_PACKAGE_PREFIX
• MINGW_PREFIX
• MINITAB_LICENSE_FILE
• MLM_LICENSE_FILE
• MSC_LICENSE_FILE
• MSIPATH1
• MSIPATH2
• MSIPATH3
• MSYSTEM
• MSYSTEM_CARCH
• MSYSTEM_CHOST
• MSYSTEM_PREFIX
• MVELD_LICENSE_FILE
• NSITECHN_LICENSE_FILE
• NUMBER_OF_PROCESSORS
• NVCUDASAMPLES9_2_ROOT
• NVCUDASAMPLES_ROOT
• ODM_LICENSE_FILE
• OLDPWD
• OPTASENSE_LICENSE_FILE
• ORIGINAL_PATH
• ORIGINAL_TEMP
• ORIGINAL_TMP
• OS
• OSP_LICENSE_FILE
• ONEDRIVE
• PALISADE_LICENSE_FILE
• PATH
• PATHEXT
• PETNEWFL_LICENSE_FILE
• PETROSYS_LICENSE_FILE
• PGROUPD_LICENSE_FILE
• PIPENV_SKIP_LOCK
• PKG_CONFIG_PATH
• PLINK_PROTOCOL
• PRDM_GEO_LICENSE_FILE
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMFILES
• PS1
• PSELMD_LICENSE_FILE
• PSG_LICENSE_FILE
• PSMODULEPATH
• PTC_D_LICENSE_FILE
• PUBLIC
• PVTLMD_LICENSE_FILE
• PWD
• PROGRAMDATA
• PROGRAMFILES(X86)
• PROGRAMW6432
• QEDILM_LICENSE_FILE
• REALCHRT_LICENSE_FILE
• RECALL5_LICENSE_FILE
• RLM_LICENSE
• ROPTIMA_LICENSE_FILE
• SAFE_LICENSE_FILE
• SBT_HOME
• SCPLMD_LICENSE_FILE
• SESSIONNAME
• SHELL
• SHLVL
• SIEPRTS_LICENSE_FILE
• SLBFD_LICENSE_FILE
• SLBSLS_LICENSE_FILE
• SNC_LIB
• SNC_LIB_64
• SPLMCS_LICENSE_FILE
• SSF_LIBRARY_PATH
• SSF_LIBRARY_PATH_64
• SSH_ASKPASS
• STACK_ROOT
• STA_FLXM_LICENSE_FILE
• STRATADATA_LICENSE
• STRSOLV_LICENSE_FILE
• SWRAP_LICENSE_FILE
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TERM
• TESSDATA_PREFIX
• THERMOCORP_LICENSE_FILE
• TMP
• TMPDIR
• TNSNAMES
• TNS_ADMIN
• TOKEN_LICENSE_FILE
• TRAMONTA_LICENSE_FILE
• UATDATA
• USERDNSDOMAIN
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VRCONTXT_LICENSE_FILE
• WINDIR
• _
• HTTP_PROXY
• HTTPS_PROXY
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv�specific environment variables:

• PIPENV_SKIP_LOCK: True

Debug�specific environment variables:

• PATH: C:\Users\Christophe.Grandsire\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\usr\bin;C:\Program Files\ConEmu\ConEmu\Scripts;C:\Program Files\ConEmu;C:\Program Files\ConEmu\ConEmu;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\libnvvp;C:\Program Files\Python37\Scripts;C:\Program Files\Python37;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;C:\Program Files\1E\NomadBranch;C:\Program Files (x86)\Plantronics\Spokes3G;C:\Program Files\MATLAB\R2018a\runtime\win64;C:\Program Files\MATLAB\R2018a\bin;C:\Program Files\dotnet;C:\Program Files\PuTTY;C:\Program Files\Java\jdk-11.0.1\bin;C:\Program Files (x86)\sbt\bin;C:\Program Files\Tesseract-OCR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\Scripts;C:\Users\Christophe.Grandsire\AppData\Roaming\local\bin;C:\Users\Christophe.Grandsire\AppData\Local\Microsoft\WindowsApps;C:\Users\Christophe.Grandsire\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Christophe.Grandsire\AppData\Local\GitHubDesktop\bin;C:\Users\Christophe.Grandsire\.virtualenvs\Global_Isopach-OUJ2C1wF\Lib\site-packages\osgeo;C:\Users\Christophe.Grandsire\Julia-1.0.2\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
• SHELL: C:\Program Files\Git\usr\bin\bash.exe
• LANG: en_US.UTF-8
• PWD: C:/Users/Christophe.Grandsire/Cases/Code/Equipment Mapping

---

Contents of Pipfile (‘C:\Users\Christophe.Grandsire\Cases\Code\Equipment Mapping\Pipfile’):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pylint = "*"
pydocstyle = "*"
rope = "*"
jupyter = "*"

[packages]
pymupdf = "*"
pdfplumber = "*"
regex = "*"
numpy = {path = "C:/Users/Christophe.Grandsire/Downloads/numpy-1.15.4-cp37-cp37m-win_amd64.whl"}
pandas = "*"
xlrd = "*"
pdfquery = "*"

[requires]
python_version = "3.7"

Thanks in advance for your help!

[tsela]

Sorry for not coming back to this earlier. Because of this and other problems I actually moved my development setup to Linux, so I cannot try what you ask. Also, I fail to see what adding --site-packages would do to help, as this issue does not concern system-wide installed packages but rather an incompatibility between Pipenv’s tendency to reinstall packages over and over even when unnecessary and Windows’s locking DLLs when in use.

[tsela]

Sorry, in the meantime I have stopped using Pipenv, so I can’t check whether this problem is solved. Maybe the other people on this thread who mentioned having the same problem can chime in, @IvanaGyro and @evilUrge.