Reopen of #3903: extras_require isn't honored if a package is already "discovered" without them

#3903

• Pip version: 10.0.0.dev0 (b9f70f206138d9a11158b411faf0329deb8d1d8b) – also reproduces with 8.1.2, 9.0.1, and other versions I’ve tried
• Python version: python2.7 – also reproduces with other versions I’ve tried (3.5, 3.6)
• Operating system: linux_x86_64 (In my case ubuntu xenial) – also reproduces on other linuxes and assumedly osx / windows as well

Description:

Installing pre-commit==0.13.0 and bravado-core should also install the dependencies of jsconschema[format] (from bravado-core) (for example webcolors)

What I’ve run:

# removing cache is  unnecessary, but for sanity's sake
rm -rf venv ~/.cache/pip
virtualenv venv
. venv/bin/activate
echo -e 'pre-commit==0.13.0\nbravado-core' > requirements.txt
pip install -r requirements.txt
# I expect this to contain webcolors
pip freeze -l

Output

$ # removing cache is  unnecessary, but for sanity's sake
$ rm -rf venv ~/.cache/pip
$ virtualenv venv
New python executable in /tmp/foo/venv/bin/python
Installing setuptools, pip, wheel...done.
$ . venv/bin/activate
(venv) $ echo -e 'pre-commit==0.13.0\nbravado-core' > requirements.txt
(venv) $ pip install -r requirements.txt
Collecting pre-commit==0.13.0 (from -r requirements.txt (line 1))
  Downloading pre_commit-0.13.0-py2.py3-none-any.whl (135kB)
    100% |████████████████████████████████| 143kB 1.4MB/s 
Collecting bravado-core (from -r requirements.txt (line 2))
  Downloading bravado-core-4.7.2.tar.gz
Collecting aspy.yaml (from pre-commit==0.13.0->-r requirements.txt (line 1))
  Downloading aspy.yaml-0.3.0-py2.py3-none-any.whl
Collecting jsonschema (from pre-commit==0.13.0->-r requirements.txt (line 1))
  Downloading jsonschema-2.6.0-py2.py3-none-any.whl
Collecting cached-property (from pre-commit==0.13.0->-r requirements.txt (line 1))
  Downloading cached_property-1.3.0-py2.py3-none-any.whl
Collecting pyyaml (from pre-commit==0.13.0->-r requirements.txt (line 1))
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% |████████████████████████████████| 256kB 879kB/s 
Collecting virtualenv (from pre-commit==0.13.0->-r requirements.txt (line 1))
  Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 387kB/s 
Collecting nodeenv>=0.11.1 (from pre-commit==0.13.0->-r requirements.txt (line 1))
  Downloading nodeenv-1.1.2.tar.gz
Collecting python-dateutil (from bravado-core->-r requirements.txt (line 2))
  Downloading python_dateutil-2.6.0-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 875kB/s 
Collecting simplejson (from bravado-core->-r requirements.txt (line 2))
  Downloading simplejson-3.10.0.tar.gz (77kB)
    100% |████████████████████████████████| 81kB 1.3MB/s 
Requirement already satisfied: six in ./venv/lib/python2.7/site-packages (from bravado-core->-r requirements.txt (line 2))
Collecting swagger-spec-validator>=2.0.1 (from bravado-core->-r requirements.txt (line 2))
  Downloading swagger-spec-validator-2.1.0.tar.gz
Collecting pytz (from bravado-core->-r requirements.txt (line 2))
  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
    100% |████████████████████████████████| 491kB 709kB/s 
Collecting functools32; python_version == "2.7" (from jsonschema->pre-commit==0.13.0->-r requirements.txt (line 1))
  Downloading functools32-3.2.3-2.zip
Building wheels for collected packages: bravado-core, pyyaml, nodeenv, simplejson, swagger-spec-validator, functools32
  Running setup.py bdist_wheel for bravado-core ... done
  Stored in directory: /home/asottile/.cache/pip/wheels/f7/e7/1f/1925a22efa17f032ac89a5b9879f391440e74a0bac1831ce66
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /home/asottile/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for nodeenv ... done
  Stored in directory: /home/asottile/.cache/pip/wheels/b8/d5/01/5e8f6e846212747b154528bfc49796590883bedb4137793b08
  Running setup.py bdist_wheel for simplejson ... done
  Stored in directory: /home/asottile/.cache/pip/wheels/43/c5/ef/edcebbb19becffd2ba75bf219afdbb4ca85198b2d909f1b31b
  Running setup.py bdist_wheel for swagger-spec-validator ... done
  Stored in directory: /home/asottile/.cache/pip/wheels/2c/ca/49/214a7376ac5ab178f350dc30e9afc4aefd1d59bd0e1da2786b
  Running setup.py bdist_wheel for functools32 ... done
  Stored in directory: /home/asottile/.cache/pip/wheels/3c/d0/09/cd78d0ff4d6cfecfbd730782a7815a4571cd2cd4d2ed6e69d9
Successfully built bravado-core pyyaml nodeenv simplejson swagger-spec-validator functools32
Installing collected packages: pyyaml, aspy.yaml, functools32, jsonschema, cached-property, virtualenv, nodeenv, pre-commit, python-dateutil, simplejson, swagger-spec-validator, pytz, bravado-core
Successfully installed aspy.yaml-0.3.0 bravado-core-4.7.2 cached-property-1.3.0 functools32-3.2.3.post2 jsonschema-2.6.0 nodeenv-1.1.2 pre-commit-0.13.0 python-dateutil-2.6.0 pytz-2017.2 pyyaml-3.12 simplejson-3.10.0 swagger-spec-validator-2.1.0 virtualenv-15.1.0
(venv) $ # I expect this to contain webcolors
(venv) $ pip freeze -l
appdirs==1.4.3
aspy.yaml==0.3.0
bravado-core==4.7.2
cached-property==1.3.0
functools32==3.2.3.post2
jsonschema==2.6.0
nodeenv==1.1.2
packaging==16.8
pre-commit==0.13.0
pyparsing==2.2.0
python-dateutil==2.6.0
pytz==2017.2
PyYAML==3.12
simplejson==3.10.0
six==1.10.0
swagger-spec-validator==2.1.0
virtualenv==15.1.0

[uranusjr]

I expect the new resolver to not have this issue. IIRC there is a test for specifically this issue 🙂

[uranusjr]

Sidenote: The title of this issue confused me for a while. I thought it’s describing a problem where an already installed package would make a subsequence pip install miss an extra (it wouldn’t, even using the legacy resolver).

The actual issue here is however that pip misses extras if the same package is requested multiple times in the same install command, but the one without extras is listed before the one with extras.