Unexplainable ContextualVersionConflict

Not sure how I am going to describe this, as I cannot find any way to reproduce this on my own machine, but for some reason, on Travis CI for Python 2.7.15, pip fails to uninstall the already present numpy v1.16.0 and install the requested numpy<1.16.0,>=1.12.0. The build job can be viewed here: https://travis-ci.com/1313e/PRISM/jobs/178432270

I know that when running on Travis CI on Linux, it automatically comes with NumPy 1.16.0, which is incompatible with what I want to test for Python 2.7. Therefore, when installing all requirements, it should uninstall that version and install a version that satisfies the condition (which is v1.15.4). Until today, it did this perfectly fine and it still does on my machine(s). However, for some odd reason, it does not do that anymore and simply throws an ContextualVersionConflict.

The only thing I changed today is that I combined two requirements files into a single one. One has a package requirement that simply requires NumPy (emcee), while the other has the version restrictions. This does work perfectly fine on my machine(s), even in fresh environments, so I am a bit confused why this pops up all of a sudden.

I know this may seem a bit of a weird issue, as I cannot give any way or method to reproduce the error, but I hope somebody knows what the problem is here.

Edit: It seems that this problem affects both pip v18.1 and v19.0.2.

[uranusjr]

I think I’ve found the problem. This happens when you try to build an egg-info with a conflicting requirement in the environment.

# setup.py
from setuptools import setup
setup(
    name='conflict-test',
    install_requires=['scripttest!=1.3'],
)

# IMPORTANT: Trigger egg-info rebuild.
rm -rf ./conflict_test.egg-info

# Build a fresh environment.
rm -rf ./venv
python3.6 -m venv venv
PYTHON="$PWD/venv/bin/python"

# Install an incompatible requirement.
$PYTHON -m pip install scripttest==1.3

# FAILURE!
$PYTHON -m pip install -e .

When running pip install, pip tries to check that the metadata is ready with InstallRequirement.check_if_exists(). For an sdist/editable (I think, please correct me if I’m wrong), this means it looks for the egg-info directory, and sets context whether the requirement is satisfied or has conflicts.

https://github.com/pypa/pip/blob/cba6083f92eb6137c623f8c99b36e6b150b41e66/src/pip/_internal/req/req_install.py#L376-L416

From what I understand, the large try-except block here follows this logic:

1. Try to find an existing, compatible distribution for the package to use.
2. If no distributions are found, good (we’ll install it).
3. If a distribution is found, but it is incompatible, record the offending distribution.

The problem is how 3. is implemented. The code assumes that if get_distribution("{package}=={version}") fails with VersionConflict, get_distribution("{package}") (dropping the specifiers) should return the correct result. But in the reported case, VersionConflict is raised not because the to-be-installed package conflicts, but one of its dependencies do. So the second get_distribution("{package}") still fails, resulting in a strange error.

I am not sure how this should be fixed. Maybe pkg_resources.get_distribution() should grow a flag so it can be used without checking conflicts in dependencies, or maybe should not use this function to get an installed distribution in the first place. Or maybe pip should uninstall that conflicting dependency (it will if this code block works anyway).

@1313e A workaround is to run python setup.py egg_info before pip install -e. This makes pip choose the 1. code path above, avoiding the bug.

[uranusjr]

Yes I was! I wasn’t able to reproduce before because my test case was either too clean or not clean enough (a prebuilt egg-info resolves the problem). Travis has an environment right in the sweet spot, and local reproduction is easy once I know where that spot is :p