pip 20.3b1 bug with --pre for already installed package

Environment

• pip version: 20.3b1
• Python version: 3.8
• OS: Linux

Description

After the release of pip 20.3b1 yesterday we started seeing issues on SciPy CI jobs that use pip install --pre --upgrade numpy. Here is a full build log: https://github.com/scipy/scipy/pull/12729#issuecomment-720111394

There’s two exceptions actually, the version normalization seems also to be missing somewhere. But it’s the first exception that’s the more problematic one.

Expected behavior

Package should be upgraded without any errors.

How to Reproduce

Here is a standalone reproducer:

conda create -n piptest python=3.8 pip=19.3  # start from 19.3 (TravisCI default)
conda activate piptest
pip install -U pip --pre  # installs 20.3b1
pip install numpy==1.17.3  # install an older numpy version, so we can upgrade
pip install --pre --upgrade numpy  -i https://pypi.anaconda.org/scipy-wheels-nightly/simple

Output

Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Requirement already satisfied: numpy in /home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages (1.17.3)
ERROR: Exception:
Traceback (most recent call last):                                                                                            
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 186, in _merge_into_criterion                                                                                                          
    crit = self.state.criteria[name]                                                                                          
KeyError: 'numpy'                                                                                                             

During handling of the above exception, another exception occurred:                                                           

Traceback (most recent call last):                                                                                            
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 217, in _main                                                                                                                            
    status = self.run(options, args)                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper                                                                                                                           
    return func(self, options, args)                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 318, in run
    requirement_set = resolver.resolve(                                                                                       
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 119, in resolve                                                                                                            
    self._result = resolver.resolve(                                                                                          
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve                                                                                                                        
    state = resolution.resolve(requirements, max_rounds=max_rounds)                                                           
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 294, in resolve                                                                                                                        
    name, crit = self._merge_into_criterion(r, parent=None)                                                                   
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 188, in _merge_into_criterion
    crit = Criterion.from_requirement(self._p, requirement, parent)
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 89, in from_requirement
    if not candidates:
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 96, in __bool__
    return any(self)
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 81, in __iter__
    candidates = _insert_installed(self._installed, self._get_others())
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 41, in _insert_installed
    candidates = sorted(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 198, in iter_index_candidates
    yield self._make_candidate_from_link(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 144, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/rgommers/anaconda3/envs/piptest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 286, in __init__
    assert str(version) == wheel.version, (
AssertionError: <Version('1.20.0.dev0+20201101041405.5c37621')> != '1.20.0.dev0+20201101041405-5c37621' for wheel numpy

[uranusjr]

By the first exception I assume you mean KeyError: 'numpy'? That’s not a bug, but an expected exception. But the exception is handled like this (very simplified):

try:
    crit = self.state.criteria[name]
except KeyError:
    crit = _calculate_criteria_differently(...) # This eventually calls the above assertion.

And that handler blew up when it shouldn’t.

[pradyunsg]

@nagesh-chowdaiah what version of pip are you using?