venv-update doesn't allow for bootstrap requirements to also be declared in requirements-dev.txt

requirements-tools isn’t aware of requirements-bootstrap.txt and can generate requirements files that include things already declared in requirements-bootstrap.txt. e.g. pre-commit->importlib-resources->wheel, where wheel is needed for bootstrapping.

venv-update will fail to install because “Double requirement given”.

• Removing wheel from requirements-bootstrap.txt makes requirement-tools happy but it means venv-update doesn’t necessarily have the right wheel installed before continuing to install rest of requirements.
• Removing wheel from requirements-dev.txt causes check-requirements to fail. That tool is valuable to ensure requirements are completely declared and pinned

[asottile]

If you upgrade to importlib-resources==1.0.2 I’ve fixed the dependence on wheel

[ipwnponies]

I’ve confirmed this is not a venv-update issue but to do with tox-pip-extensions. Here are the files I’m using to set up repro, with updated pip and venv-update versions:

$ ls
requirements-bootstrap.txt  requirements.txt  tox.ini  venv_update.py
$ tail requirements*
==> requirements-bootstrap.txt <==
pip==18.1
pip-custom-platform==0.5.0
pymonkey==0.3.0
venv-update==3.2.2
wheel==0.26.0

==> requirements.txt <==
wheel

venv-update works correctly. Note that it installs wheel==0.26.0 during bootstrap, then upgrades to wheel==0.33.1 when installing requirements. Which is exactly what we want (bootstrap requirements independent from application requirements), therefore the use case I mentioned earlier is already handled properly and this issue is invalid.

$ python3 venv_update.py bootstrap-deps= -r requirements-bootstrap.txt install= -r requirements.txt
> virtualenv venv
Using real prefix '/usr/local/Cellar/python/3.7.2_1/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/jngu/venv-update-issue/venv/bin/python
Installing setuptools, pip, wheel...
done.
> rm -rf venv/local
> pip install -r requirements-bootstrap.txt
Collecting pip==18.1 (from -r requirements-bootstrap.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl
Collecting pip-custom-platform==0.5.0 (from -r requirements-bootstrap.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/02/37/b60c96cc0ef007eee8db554da9a190ca2f366a136600fbf8829cbad19563/pip_custom_platform-0.5.0-py2.py3-none-any.whl
Collecting pymonkey==0.3.0 (from -r requirements-bootstrap.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/da/a9/1a74cf32dfb17eb7db7941174ab2c47d54b2d1f018dbd02ae7de9d2047b7/pymonkey-0.3.0-py2.py3-none-any.whl
Collecting venv-update==3.2.2 (from -r requirements-bootstrap.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/1f/0a/89e39ec9b211f42e4b8a14a1bee289a001479b23ad546d812c3720ed9cef/venv_update-3.2.2-py2.py3-none-any.whl
Collecting wheel==0.26.0 (from -r requirements-bootstrap.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/d9/6f/cfb45d489703e4843b30c28e01ab39c87b0648bfa11a6c4a959f426fe217/wheel-0.26.0-py2.py3-none-any.whl
Collecting distro>=1.2.0 (from pip-custom-platform==0.5.0->-r requirements-bootstrap.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=0.8.0 in ./venv/lib/python3.7/site-packages (from venv-update==3.2.2->-r requirements-bootstrap.txt (line 4)) (40.8.0)
Installing collected packages: pip, pymonkey, wheel, distro, pip-custom-platform, venv-update
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
  Found existing installation: wheel 0.33.1
    Uninstalling wheel-0.33.1:
      Successfully uninstalled wheel-0.33.1
Successfully installed distro-1.4.0 pip-18.1 pip-custom-platform-0.5.0 pymonkey-0.3.0 venv-update-3.2.2 wheel-0.26.0
> pip-faster install --upgrade --prune -r requirements.txt
slow: full search for unpinned requirement wheel in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 1))
Collecting wheel (from -r requirements.txt (line 1))
  slow: full search for unpinned requirement wheel (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/96/ba/a4702cbb6a3a485239fbe9525443446203f00771af9ac000fa3ef2788201/wheel-0.33.1-py2.py3-none-any.whl
Installing collected packages: wheel
  Found existing installation: wheel 0.26.0
    Uninstalling wheel-0.26.0:
      Successfully uninstalled wheel-0.26.0
Successfully installed wheel-0.33.1
> pip uninstall --yes distro pip-custom-platform pymonkey
Uninstalling distro-1.4.0:
  Successfully uninstalled distro-1.4.0
Uninstalling pip-custom-platform-0.5.0:
  Successfully uninstalled pip-custom-platform-0.5.0
Uninstalling pymonkey-0.3.0:
  Successfully uninstalled pymonkey-0.3.0

I was able to reproduce issue under tox environment, I’ll open a ticket on tox-pip-extension and take discussion there.