Test test_hashed_install_failure_later_flag is trying to check too much in one test

Environment

• pip version: 20.1.1
• Python version: Not version-specific
• OS: Not OS-specific

Description The test test_hashed_install_failure_later_flag, in test_install.py, sets up a variety of hash checking issues, and then checks that pip reports a very specific error message, covering all of the problems that were set up. This is problematic, because the test is very dependent on the precise wording of the message pip produces, as well as not isolating specific problems.

Expected behavior We should have one test per issue, rather than a single “do everything at once” test.

Additional context Pip’s new resolver produces different error message text for many of the errors that arise during resolution, either from simply rewording things for clarity, or because the existing wording doesn’t correctly reflect how the new resolver works. So tests that check irrelevant details of what an error message says are difficult to handle during the transition period when both resolvers are available. Tests that check functionality, rather than reporting, are more robust.

Source code The code for the test as at the time of writing is included below, as the test will probably be modified for the new resolver.

@pytest.mark.network
@pytest.mark.fails_on_new_resolver
def test_hashed_install_failure_later_flag(script, tmpdir):
    with requirements_file(
        "blessings==1.0\n"
        "tracefront==0.1 --hash=sha256:somehash\n"
        "https://files.pythonhosted.org/packages/source/m/more-itertools/"
        "more-itertools-1.0.tar.gz#md5=b21850c3cfa7efbb70fd662ab5413bdd\n"
        "https://files.pythonhosted.org/"
        "packages/source/p/peep/peep-3.1.1.tar.gz\n",
        tmpdir,
    ) as reqs_file:
        result = script.pip(
            "install", "-r", reqs_file.resolve(), expect_error=True
        )

    assert_re_match(
        r'Hashes are required in --require-hashes mode, but they are '
        r'missing .*\n'
        r'    https://files\.pythonhosted\.org/packages/source/p/peep/peep'
        r'-3\.1\.1\.tar\.gz --hash=sha256:[0-9a-f]+\n'
        r'    blessings==1.0 --hash=sha256:[0-9a-f]+\n'
        r'THESE PACKAGES DO NOT MATCH THE HASHES.*\n'
        r'    tracefront==0.1 .*:\n'
        r'        Expected sha256 somehash\n'
        r'             Got        [0-9a-f]+',
        result.stderr,
    )

[pradyunsg]

I think that test is correct, and the aggregation behavior it is checking is something we should consider implementing in the new resolver.

OTOH, I’m 100% OK with letting this test fail as-is for now, until after the beta is out, to see if anyone says anything during the beta about wanting this functionality. If no one expresses interest in this, we can skip the aggregation behavior and break this test up.

[uranusjr]

Looking at the name of the test, I kind of feel that having multiple hash errors is actually the thing the test wants to ensure. For each hash error, the (legacy) resolver raises a HashError, and catches all of them “later” to raise an aggregated HashErrors. Having multiple hash errors in one pip command would be the design goal if that’s a correct assessment.