Pip doesn’t allow you to self-depend

Description

When specifying dependencies, you sometimes want to depend on other extras:

[project]
name = 'my_pkg'

[project.optional-dependencies]
test-utils = ['pytest', 'requests-mock', '...']
test = ['my_pkg[test-utils]']

Expected behavior

@uranusjr said it should work:

Circular dependency is a feature that Python packaging is explicitly designed to allow, so it works and should continue to work. […]

pip version

21.2.4

Python version

3.9.6

OS

Arch Linux

How to Reproduce

1. Create a new package with something like the above setup
2. Install it with the extra depending on itself: pip install .[test]
3. It can’t find the package because it doesn’t merge . with my_pkg

Specifying test = ['.[test-utils]'] doesn’t work.

Output

@henryiii said:

It does not pull the extra from the current package, but rather from the existing cached packages. So if you add a new extra […] and then depend on it, it starts spewing out a long list of package checking:

Collecting my_pkg[test-utils]
  Using cached my_pkg-1.11.1-py3-none-any.whl (1.5 MB)
WARNING: my_pkg 1.11.1 does not provide the extra 'test-utils'
  Using cached my_pkg-1.11.0-py3-none-any.whl (1.5 MB)
WARNING: my_pkg 1.11.0 does not provide the extra 'test-utils'
  Using cached my_pkg-1.10.0-py3-none-any.whl (1.5 MB)
WARNING: my_pkg 1.10.0 does not provide the extra 'test-utils'

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[henryiii]

Okay, results:

• pip 20.2 and before simply ignore the extras (only testing 20.2, since I’m on macOS 11)
• pip 20.3 to 21.1 fail by looking at all older, online versions
• pip 21.2 & 21.3 work correctly.

import nox

@nox.session
@nox.parametrize('pip', ['20.2.*', '20.3.*', '21.0.*', '21.1.*', '21.2.*', '21.3.*'])
def tests(session: nox.Session, pip: str) -> None:
    session.install(f"pip=={pip}")
    session.run("pip", "--version")
    tmp = session.create_tmp()
    session.cd(tmp)

    with open("pyproject.toml", "w") as f:
        f.write("""\
[build-system]
requires = ['flit-core']
build-backend = "flit_core.buildapi"

[project]
name = 'cibuildwheel'
version = "1"
description = "..."

[project.optional-dependencies]
test-utils = ['pytest', 'requests-mock']
test-combined = ['cibuildwheel[test-utils]']
""")
    with open("cibuildwheel.py", "w") as f:
        f.write("__version__ = '1.0.0'")

    session.install(".[test-combined]")
    session.run("python", "-c", "import pytest")

nox > * tests(pip='20.2.*'): failed
nox > * tests(pip='20.3.*'): failed
nox > * tests(pip='21.0.*'): failed
nox > * tests(pip='21.1.*'): failed
nox > * tests(pip='21.2.*'): success
nox > * tests(pip='21.3.*'): success

[uranusjr]

20.3 was when the default resolver switch, so that behavioural change makes sense. I can’t quite point out what exactly changed in 21.2 that makes this work (that release contains quite several resolver tweaks), but glad to hear its working now. I think this would be a very big step toward more declarative metadata definition since we no longer need to either write custom tooling to preprocess metadata, or manually constructing extras in setup.py to avoid duplication.