Pipenv reinstalling all dependencies from Pipfile when installing a new package causes issues on Windows
See original GitHub issueIssue description
When installing a new package via Pipenv, after installing that new package Pipenv somehow reinstalls all packages mentioned in the Pipfile (as mentioned by the line "Installing dependencies from Pipfile), even those unrelated to that new package. This may work fine on Unix-based OSes, but on Windows there is that problem that Windows locks access to files that are currently in use by another process, so for instance when trying to install a new package while running a Jupyter notebook, the process will end up failing because the reinstalling step tried to overwrite a locked file, resulting in a “[WinError 5] Access is denied” error.
Note that the new package is usually installed correctly, and normally nothing gets actually broken, but it is annoying, especially since there is no reason why installing a new package should force a reinstall of all packages in the virtual environment, especially the unrelated ones. It makes the process much slower than it could be, and it causes the kind of issues I described above.
Expected result
I have a virtual environment containing a few packages including PyMuPDF (an independent PDF-handling library with no dependencies and no dependents), and a Jupyter notebook running that makes use of PyMuPDF. I want to test another PDF library, PDFQuery, which is independent from PyMuPDF. If I do pipenv install pdfquery
, my expectation is for Pipenv to install it, its dependencies if they are not already present, and I should get a message that the installation has succeeded.
Actual result
Instead, here is the result:
$ pipenv install pdfquery
Installing pdfquery�
Adding pdfquery to Pipfile's [packages]�
Installation Succeeded
Installing dependencies from Pipfile�
An error occurred while installing pymupdf! Will try again.
An error occurred while installing C:/Users/Christophe.Grandsire/Downloads/numpy-1.15.4-cp37-cp37m-win_amd64.whl! Will try again.
Installing initially failed dependencies�
[pipenv.exceptions.InstallError]: File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\cli\command.py", line 254, in install
[pipenv.exceptions.InstallError]: editable_packages=state.installstate.editables,
[pipenv.exceptions.InstallError]: File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]: skip_lock=skip_lock,
[pipenv.exceptions.InstallError]: File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]: pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]: File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 862, in do_install_dependencies
[pipenv.exceptions.InstallError]: _cleanup_procs(procs, False, failed_deps_queue, retry=False)
[pipenv.exceptions.InstallError]: File "C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\site-packages\pipenv\core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]: raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting pymupdf', ' Using cached https://files.pythonhosted.org/packages/f0/45/b5a90538a9603708dfc705d7d318f4192a5ffcd4c41b360b80624040fe6c/PyMuPDF-1.14.4-cp37-cp37m-win_amd64.whl', 'Installing collected packages: pymupdf', ' Found existing installation: PyMuPDF 1.14.3', ' Uninstalling PyMuPDF-1.14.3:', '
Successfully uninstalled PyMuPDF-1.14.3']
[pipenv.exceptions.InstallError]: ["Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\\\Users\\\\CHRIST~1.GRA\\\\AppData\\\\Local\\\\Temp\\\\pip-uninstall-adfcfhxg\\\\users\\\\christophe.grandsire\\\\.virtualenvs\\\\equipment_mapping-lnlbcdyh\\\\lib\\\\site-packages\\\\fitz\\\\_fitz.pyd'", 'Consider using the `--user` option or check the permissions.']
ERROR: ERROR: Package installation failed...
As you can see, installing PDFQuery prompted a reinstall of PyMuPDF (despite the two packages being completely unrelated. They don’t even have common dependencies), but since PyMuPDF is currently in use in my open Jupyter notebook, the _fitz.pyd
file (a part of PyMuPDF) is locked by Windows, and the attempt to overwrite it causes the [pipenv.exceptions.InstallError]: ["Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\\\Users\\\\CHRIST~1.GRA\\\\AppData\\\\Local\\\\Temp\\\\pip-uninstall-adfcfhxg\\\\users\\\\christophe.grandsire\\\\.virtualenvs\\\\equipment_mapping-lnlbcdyh\\\\lib\\\\site-packages\\\\fitz\\\\_fitz.pyd'", 'Consider using the
–user option or check the permissions.']
, and the following ERROR: ERROR: Package installation failed...
, which is particularly egregious since PDFQuery was actually installed correctly and works fine.
Steps to replicate
I can’t provide a full list of steps to replicate as it would require me to provide the Jupyter notebook I’m using and the data it relies on, and I’m not allowed to share them outside my company. But it should be possible to reproduce this error on any Windows computer (I’m on Windows 10) with a virtual environment containing a specific package currently in use by a Jupyter notebook, and then try to install any other package in that environment. You may need to make sure that specific package contains a .pyd
file, which is basically a Windows DLL and is probably the reason why Windows refuses to overwrite it.
Here is the output of pipenv --support
:
$ pipenv --support
Pipenv version: '2018.11.26'
Pipenv location: 'C:\\Users\\Christophe.Grandsire\\AppData\\Roaming\\Python\\Python37\\site-packages\\pipenv'
Python location: 'c:\\program files\\python37\\python.exe'
Python installations found:
3.7.0
:C:\Program Files\Python37\python.exe
PEP 508 Information:
{'implementation_name': 'cpython',
'implementation_version': '3.7.0',
'os_name': 'nt',
'platform_machine': 'AMD64',
'platform_python_implementation': 'CPython',
'platform_release': '10',
'platform_system': 'Windows',
'platform_version': '10.0.16299',
'python_full_version': '3.7.0',
'python_version': '3.7',
'sys_platform': 'win32'}
System environment variables:
ABAQUSLM_LICENSE_FILE
ACLOCAL_PATH
ADSKFLEX_LICENSE_FILE
AD_PATH
ALLUSERSPROFILE
ALT_LICENSE_FILE
ANSICON
ANSICON_DEF
APPDATA
BEICIP_LICENSE_FILE
BLM_LICENSE_FILE
BLUECIEL_LICENSE_FILE
BMGEO_LICENSE_FILE
CAESAR_LICENSE_FILE
CALLMD_LICENSE_FILE
CDLMD_LICENSE_FILE
CGLD_LICENSE_FILE
CMG_LIC_HOST
COMMONPROGRAMFILES
COMPUTERNAME
COMSPEC
CONFIG_SITE
CUDA_PATH
CUDA_PATH_V9_2
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
CONEMUANSI
CONEMUANSILOG
CONEMUARGS2
CONEMUARGS
CONEMUBACKHWND
CONEMUBASEDIR
CONEMUBASEDIRSHORT
CONEMUBUILD
CONEMUCFGDIR
CONEMUCONFIG
CONEMUDIR
CONEMUDRAWHWND
CONEMUDRIVE
CONEMUHWND
CONEMUHOOKS
CONEMUPID
CONEMUPALETTE
CONEMUSERVERPID
CONEMUTASK
CONEMUWORKDIR
CONEMUWORKDRIVE
DHSDELFT_LICENSE_FILE
DISPLAY
DNVSLM_LICENSE_FILE
ELIIS_LICENSE_FILE
ENRESINT_LICENSE_FILE
EXEPATH
FLEXLM_NO_CKOUT_INSTALL_LIC
GDAL_DRIVER_PATH
GEOCOSM_LICENSE_FILE
GEOMATICLM_LICENSE_FILE
GEOSOFT_LICENSE_FILE
GEOTOMO_LICENSE_FILE
HAMP_RUSS_LICENSE_FILE
HOME
HOMEDRIVE
HOMEPATH
HOMESHARE
HOSTNAME
HPQ_LICENSE_FILE
HRH_LICENSE_FILE
IDL_LMGRD_LICENSE_FILE
IHS_LICENSE_FILE
IKON_LICENSE_FILE
IMSTATUS
INFOPATH
INTEL_LICENSE_FILE
IPASSI_LICENSE_FILE
IVS_LICENSE_FILE
JASONLM_LICENSE_FILE
JAVA_HOME
JOA_LICENSE_FILE
JUSTINT_LICENSE_FILE
KAPPA_LICENSE_FILE
KNOWSYS_LICENSE_FILE
LANG
LCT_LMD_LICENSE_FILE
LGCX_LICENSE_FILE
LICSRV_LICENSE_FILE
LMGRD.SLB_LICENSE_FILE
LM_PROJECT
LOCALAPPDATA
LOGONSERVER
LSHOST
MANPATH
MARINLMD_LICENSE_FILE
MCSLMD_LICENSE_FILE
MDAXBBWC_LICENSE_FILE
MFRACLM_LICENSE_FILE
MINGW_CHOST
MINGW_PACKAGE_PREFIX
MINGW_PREFIX
MINITAB_LICENSE_FILE
MLM_LICENSE_FILE
MSC_LICENSE_FILE
MSIPATH1
MSIPATH2
MSIPATH3
MSYSTEM
MSYSTEM_CARCH
MSYSTEM_CHOST
MSYSTEM_PREFIX
MVELD_LICENSE_FILE
NSITECHN_LICENSE_FILE
NUMBER_OF_PROCESSORS
NVCUDASAMPLES9_2_ROOT
NVCUDASAMPLES_ROOT
ODM_LICENSE_FILE
OLDPWD
OPTASENSE_LICENSE_FILE
ORIGINAL_PATH
ORIGINAL_TEMP
ORIGINAL_TMP
OS
OSP_LICENSE_FILE
ONEDRIVE
PALISADE_LICENSE_FILE
PATH
PATHEXT
PETNEWFL_LICENSE_FILE
PETROSYS_LICENSE_FILE
PGROUPD_LICENSE_FILE
PIPENV_SKIP_LOCK
PKG_CONFIG_PATH
PLINK_PROTOCOL
PRDM_GEO_LICENSE_FILE
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMFILES
PS1
PSELMD_LICENSE_FILE
PSG_LICENSE_FILE
PSMODULEPATH
PTC_D_LICENSE_FILE
PUBLIC
PVTLMD_LICENSE_FILE
PWD
PROGRAMDATA
PROGRAMFILES(X86)
PROGRAMW6432
QEDILM_LICENSE_FILE
REALCHRT_LICENSE_FILE
RECALL5_LICENSE_FILE
RLM_LICENSE
ROPTIMA_LICENSE_FILE
SAFE_LICENSE_FILE
SBT_HOME
SCPLMD_LICENSE_FILE
SESSIONNAME
SHELL
SHLVL
SIEPRTS_LICENSE_FILE
SLBFD_LICENSE_FILE
SLBSLS_LICENSE_FILE
SNC_LIB
SNC_LIB_64
SPLMCS_LICENSE_FILE
SSF_LIBRARY_PATH
SSF_LIBRARY_PATH_64
SSH_ASKPASS
STACK_ROOT
STA_FLXM_LICENSE_FILE
STRATADATA_LICENSE
STRSOLV_LICENSE_FILE
SWRAP_LICENSE_FILE
SYSTEMDRIVE
SYSTEMROOT
TEMP
TERM
TESSDATA_PREFIX
THERMOCORP_LICENSE_FILE
TMP
TMPDIR
TNSNAMES
TNS_ADMIN
TOKEN_LICENSE_FILE
TRAMONTA_LICENSE_FILE
UATDATA
USERDNSDOMAIN
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
VRCONTXT_LICENSE_FILE
WINDIR
_
HTTP_PROXY
HTTPS_PROXY
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED
Pipenv�specific environment variables:
PIPENV_SKIP_LOCK
:True
Debug�specific environment variables:
PATH
:C:\Users\Christophe.Grandsire\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\usr\bin;C:\Program Files\ConEmu\ConEmu\Scripts;C:\Program Files\ConEmu;C:\Program Files\ConEmu\ConEmu;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\libnvvp;C:\Program Files\Python37\Scripts;C:\Program Files\Python37;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;C:\Program Files\1E\NomadBranch;C:\Program Files (x86)\Plantronics\Spokes3G;C:\Program Files\MATLAB\R2018a\runtime\win64;C:\Program Files\MATLAB\R2018a\bin;C:\Program Files\dotnet;C:\Program Files\PuTTY;C:\Program Files\Java\jdk-11.0.1\bin;C:\Program Files (x86)\sbt\bin;C:\Program Files\Tesseract-OCR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Users\Christophe.Grandsire\AppData\Roaming\Python\Python37\Scripts;C:\Users\Christophe.Grandsire\AppData\Roaming\local\bin;C:\Users\Christophe.Grandsire\AppData\Local\Microsoft\WindowsApps;C:\Users\Christophe.Grandsire\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Christophe.Grandsire\AppData\Local\GitHubDesktop\bin;C:\Users\Christophe.Grandsire\.virtualenvs\Global_Isopach-OUJ2C1wF\Lib\site-packages\osgeo;C:\Users\Christophe.Grandsire\Julia-1.0.2\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
SHELL
:C:\Program Files\Git\usr\bin\bash.exe
LANG
:en_US.UTF-8
PWD
:C:/Users/Christophe.Grandsire/Cases/Code/Equipment Mapping
Contents of Pipfile
(‘C:\Users\Christophe.Grandsire\Cases\Code\Equipment Mapping\Pipfile’):
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
pylint = "*"
pydocstyle = "*"
rope = "*"
jupyter = "*"
[packages]
pymupdf = "*"
pdfplumber = "*"
regex = "*"
numpy = {path = "C:/Users/Christophe.Grandsire/Downloads/numpy-1.15.4-cp37-cp37m-win_amd64.whl"}
pandas = "*"
xlrd = "*"
pdfquery = "*"
[requires]
python_version = "3.7"
Thanks in advance for your help!
Issue Analytics
- State:
- Created 5 years ago
- Reactions:5
- Comments:6 (1 by maintainers)
Top GitHub Comments
Sorry for not coming back to this earlier. Because of this and other problems I actually moved my development setup to Linux, so I cannot try what you ask. Also, I fail to see what adding
--site-packages
would do to help, as this issue does not concern system-wide installed packages but rather an incompatibility between Pipenv’s tendency to reinstall packages over and over even when unnecessary and Windows’s locking DLLs when in use.Sorry, in the meantime I have stopped using Pipenv, so I can’t check whether this problem is solved. Maybe the other people on this thread who mentioned having the same problem can chime in, @IvanaGyro and @evilUrge.