Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

packaging does not expect 'extra' in marker (need vendor upgrade)

See original GitHub issue

Description

Hi, I’ve been doing some hacks on pip recently, like taking extra requirements into account when checking_install_conflicts. I have rewritten the method of pkg_resources.Distribution, and every time I get the dependency, I will return the extra mim requirements. Here are the sample code:

from contextlib import contextmanager
from pip._internal.commands import create_command


@contextmanager
def patch_pkg_resources_distribution() -> Generator:
    from pip._vendor.pkg_resources import Distribution, parse_requirements

    origin_requires = Distribution.requires

    def patched_requires(self, extras=()):
        deps = origin_requires(self, extras)
        if 'mim' in self.extras:
            mim_extra_requires = origin_requires(self, ('mim',))
            filter_invalid_marker(mim_extra_requires)
            deps += mim_extra_requires
        return deps

    Distribution.requires = patched_requires  # type: ignore
    yield

    Distribution.requires = origin_requires  # type: ignore
    return


install_args = [...]
with patch_pkg_resources_distribution():
    create_command('install')(install_args)

I got this error during check_install_conflicts:

ERROR: Error while checking for conflicts. Please file an issue on pip's issue tracker: https://github.com/pypa/pip/issues/new
Traceback (most recent call last):
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 531, in _determine_conflicts
    return check_install_conflicts(to_install)
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 111, in check_install_conflicts
    package_set, should_ignore=lambda name: name not in whitelist
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 78, in check_package_set
    missed = req.marker.evaluate()
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_vendor/packaging/markers.py", line 304, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_vendor/packaging/markers.py", line 234, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_vendor/packaging/markers.py", line 216, in _get_env
    f"{name!r} does not exist in evaluation environment."
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

The reason for the above error is that this code does not consider that the marker of the requirement obtained from METADATA may contain extras. Is this behavior a bug?

A sample METADTA that contains Requires-Dist:

...
Requires-Dist: matplotlib
Requires-Dist: mmcls (>=0.20.1)
Requires-Dist: numpy
Requires-Dist: packaging
Requires-Dist: prettytable
Provides-Extra: all
Requires-Dist: cityscapesscripts ; extra == 'all'
Requires-Dist: matplotlib ; extra == 'all'
Requires-Dist: mmcls (>=0.20.1) ; extra == 'all'
Requires-Dist: numpy ; extra == 'all'
Requires-Dist: packaging ; extra == 'all'
Requires-Dist: prettytable ; extra == 'all'
Requires-Dist: codecov ; extra == 'all'
Requires-Dist: flake8 ; extra == 'all'
Requires-Dist: interrogate ; extra == 'all'
Requires-Dist: pytest ; extra == 'all'
Requires-Dist: xdoctest (>=0.10.0) ; extra == 'all'
Requires-Dist: yapf ; extra == 'all'
Provides-Extra: build
Provides-Extra: mim
Requires-Dist: mmcls (>=0.20.1) ; extra == 'mim'
Requires-Dist: mmcv-full (<=1.6.0,>=1.4.4) ; extra == 'mim'
Provides-Extra: optional
Requires-Dist: cityscapesscripts ; extra == 'optional'
Provides-Extra: tests
Requires-Dist: codecov ; extra == 'tests'
Requires-Dist: flake8 ; extra == 'tests'
Requires-Dist: interrogate ; extra == 'tests'
Requires-Dist: pytest ; extra == 'tests'
Requires-Dist: xdoctest (>=0.10.0) ; extra == 'tests'
Requires-Dist: yapf ; extra == 'tests'
...

Expected behavior

The requirement parsed from METADATA should consider the invalid situation of the marker? For example, in src/pip/_internal/operations/check.py#L78 we can handle the invalid marker?

...
for req in package_detail.dependencies:
    name = canonicalize_name(req.name)
    
    # Check if it's missing
    if name not in package_set:
        missed = True
        if req.marker is not None:
            try:
                missed = req.marker.evaluate()
            except:
                missed = False
            if missed:
                missing_deps.add((name, req))
        continue
    ....