Allow extras in constraints

Description

Currently pip does not allow extras in constraints. It did, previously. The fact that it does not now is incorrect and should be treated as a bug.

Expected behavior

Both compilations given in the steps to reproduce succeed, producing files app.txt and tests.txt.

pip version

22.3.1

Python version

3.11.0

OS

macOS 13.0.1

How to Reproduce

Create a new venv:

python3 -m venv constraint_bug
source constraint_bug/bin/activate
python3 -m pip install --upgrade pip setuptools wheel pip-tools

For me, at time of writing, this yields:

$ python3 -m pip list
Package    Version
---------- -------
build      0.9.0
click      8.1.3
packaging  21.3
pep517     0.13.0
pip        22.3.1
pip-tools  6.10.0
pyparsing  3.0.9
setuptools 65.5.1
wheel      0.38.4

Create the following files with the following contents:

app.in contains

httpx==0.23.0

tests.in contains

-c app.txt
pytest--7.2.0

And now attempt to compile with pip-tools:

$ python3 -m piptools compile --resolver=backtracking --output-file app.txt app.in
$ python3 -m piptools compile --resolver=backtracking --output-file tests.txt tests.in

Output

/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/pip/_internal/req/req_install.py:866: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
  deprecated(
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/piptools/__main__.py", line 19, in <module>
    cli()
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/piptools/scripts/compile.py", line 555, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/piptools/resolver.py", line 593, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/piptools/resolver.py", line 625, in _do_resolve
    resolver.resolve(
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james.bennett/.pyenv/versions/3.11.0/envs/constraint_bug/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 481, in collect_root_requirements
    raise InstallationError(problem)
pip._internal.exceptions.InstallationError: Constraints cannot have extras

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

The main reason extras are banned from constraints is it has unclear semantics. Say I have foo[bar] as a constraint; does it mean pip install foo should automatically enable the extra bar? Or does it simply mean foo[bar], if specified, should use the constrained version? Or even something else? Neither way is perfect and arguably both are wrong.

Your input does not contain extras, and pip-tools is generating those entries with extras. It should instead flatten it (with a semantic it chooses) before handing the content to pip.

[pfmoore]

I wanna push back on this @pfmoore – I don’t think the only way to get this feature is by filing a PR. I think making a reasonable case for what this feature could actually behave as is also a good way to move this forward.

Sorry, you’re correct. I was thinking more that to present a convincing case would need a good understanding of the current resolver and how the new constraints implementation interacts with it. I tend to conflate getting that sort of understanding with “have a go at writing a PR”.