New resolver: Failure when package only specified with extras is not available from index

What did you want to do?

pip install --use-feature=2020-resolver -r requirements.txt --no-deps pip install --use-feature=2020-resolver -r requirements.txt

requirements.txt only contains a list of packages to be installed in editable mode, with some depending on each other.

This is in a fresh miniconda environment and occurred on both 20.2.2 and master.

I ran pip using --no-deps first since in my experience, installing multiple editable mode packages with dependencies on each other fails otherwise. However, running just the normal install command directly in a fresh environment still fails with the new resolver, as below.

ERROR: Could not find a version that satisfies the requirement azureml-dataset-runtime[fuse]~=0.1.0.0 (from azureml-defaults)
ERROR: No matching distribution found for azureml-dataset-runtime[fuse]~=0.1.0.0

Output

This output is after running the 2nd pip install command, to actually install the package dependencies after installing the editable mode packages using --no-deps. Output has been slightly edited to remove full file paths.

ERROR: Cannot install azureml-dataset-runtime 0.1.0.0 (from src\azureml-dataset-runtime), -r requirements.txt (line 9), -r requirements.txt (line 16) and azureml-dataset-runtime[fuse] 0.1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested azureml-dataset-runtime 0.1.0.0 (from src\azureml-dataset-runtime)
    azureml-automl-core 0.1.0.0 depends on azureml-dataset-runtime~=0.1.0.0
    azureml-train-automl-client 0.1.0.0 depends on azureml-dataset-runtime~=0.1.0.0
    azureml-dataset-runtime[fuse] 0.1.0.0 depends on azureml-dataset-runtime 0.1.0.0 (Installed)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Additional information

The requirements file looks like this (all below packages should be available from pypi as well):

-e "src\azureml-core\."
-e "src\azureml-dataset-runtime\."
-e "src\azureml-defaults\."
-e "src\azureml-telemetry\."
-e "src\azureml-opendatasets\."
-e "src\azureml-pipeline\."
-e "src\azureml-pipeline-core\."
-e "src\azureml-pipeline-steps\."
-e "src\azureml-automl-core\."
-e "src\azureml-automl-runtime\."
-e "src\azureml-interpret\."
-e "src\azureml-explain-model\."
-e "src\azureml-train-restclients-hyperdrive\."
-e "src\azureml-train-core\."
-e "src\azureml-train\."
-e "src\azureml-train-automl-client\."
-e "src\azureml-train-automl-runtime\."

[uranusjr]

Sorry for the delay. The issue happens when pip sees a direct URL requirement, and then a requirement of the same name with different extras. Since a requirment with different extras represent a different set of dependencies, the resolver treats these two as different requirements, and is not able to resolve the non-URL requirement into using that URL.

The workaround to this would be to install the editable/URL requirements first, like this:

$ pip install --use-feature=2020-resolver -r requirements.txt --no-deps
$ pip install --use-feature=2020-resolver -r requirements.txt

The first --no-deps installation would skip all the resolver stuff (thus avoid the error). Once all those packages are in site-packages, subsequent install calls would work correctly.

Some additional technical context that might make more sense to those into pip internals: Since a[x] provides a different set of packages from a, and that set may be different depending on the version of a chosen, the resolver internally creates a “fake” package that depends on the dependencies specified by extra x, and a that matches the same version of itself. This makes pip aware of the correct version of a when it sees a[x]. But this does not work the other way around—if you provide a first, the resolver does not know it needs a[x] when it sees a (it can’t just pull it in, otherwise the user would get unwanted packages when they don’t want a[x] but only a), and we don’t have a good way to inform the resolver about that direct URL can also be used to satisfy a[x] later on. This is why installing the packages without dependencies would work around the issue. The on-disk .dist-info can be used to represent that a package, so we don’t need to use the URL.

Ultimately, this is one of those issues that are probably fixable, but require too much resource that are more useful spent elsewhere. I would be happy to offer advices if anyone really wants to work on this, but am not personally very motivated to do the work myself.

[uranusjr]

I have merged #8939, #9143, and #9204 here, which all have the same root cause, described above (https://github.com/pypa/pip/issues/8785#issuecomment-678885871).