Pip fails to find its own Exception file (21.1)

Description

At work, my team uses Github Actions to test PRs prior to their approval. When running our tests on 21.0.1 (the last time this ran was on Friday afternoon), everything was able to build and test appropriately. On Monday morning (Github now using 21.1), nothing would build, throwing the Traceback listed below.

Github as able to build Python (3.7.9 for us) and upgrade Pip. But when it came to running the first install, it failed. All Traceback is pasted below, since none of our code seemed to trigger (if curious, this was running a pre-commit hook).

Traceback errors:

Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.7.9/x64/bin/pip", line 5, in <module>
      from pip._internal.cli.main import main
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/cli/main.py", line 8, in <module>
      from pip._internal.cli.autocompletion import autocomplete
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
      from pip._internal.cli.main_parser import create_main_parser
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
      from pip._internal.cli import cmdoptions
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/cli/cmdoptions.py", line 22, in <module>
      from pip._internal.cli.progress_bars import BAR_TYPES
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/cli/progress_bars.py", line 9, in <module>
      from pip._internal.utils.logging import get_indentation
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/utils/logging.py", line 14, in <module>
      from pip._internal.utils.misc import ensure_dir
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 29, in <module>
      from pip._internal.locations import get_major_minor_version, site_packages, user_site
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/locations/__init__.py", line 9, in <module>
      from . import _distutils, _sysconfig
    File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/locations/_sysconfig.py", line 8, in <module>
      from pip._internal.exceptions import InvalidSchemeCombination, UserInstallationInvalid
  ImportError: cannot import name 'InvalidSchemeCombination' from 'pip._internal.exceptions' (/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/pip/_internal/exceptions.py)
Error: The process '/opt/hostedtoolcache/Python/3.7.9/x64/bin/pip' failed with exit code 1

Expected behavior

I expected our installation of the pre-commit hook to run without throwing any error (or if it had, only throw errors based on our code).

pip version

21.1

Python version

3.7.9

OS

Github Actions

How to Reproduce

I’m not exactly sure how to repro since this occurs on Github using a custom pre-commit file to install

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[veloxo]

Also ran into this issue when restoring a ${{ env.pythonLocation }} cache in a Github Action workflow. This isn’t a pip issue, but naively looks like one, so I’ll post a potential solution for others who may end up here like I did.

While a cache key with ${{ env.pythonLocation }} captures the Python version, the setup-python action may provide upgraded versions of Python or pip. This recent pip upgrade plus an old cache seems to result in a broken pip install.

Instead of caching the whole Python dir, try just caching installed packages while excluding pip and setuptools:

    - uses: actions/cache@v2
      with:
        # Cache the Python package environment, excluding pip and setuptools installed by setup-python
        path: |
          ~/.cache/pip
          ${{ env.pythonLocation }}/bin/*
          ${{ env.pythonLocation }}/include
          ${{ env.pythonLocation }}/lib/python*/site-packages/*
          !${{ env.pythonLocation }}/bin/pip*
          !${{ env.pythonLocation }}/lib/python*/site-packages/pip*
          !${{ env.pythonLocation }}/lib/python*/site-packages/setuptools*
          !${{ env.pythonLocation }}/lib/python*/site-packages/wheel*

This “works on my ~machine~ GIthub workflow.”

Including ~/.cache/pip helps address spuriously-excluded packages like pip-tools, while still shortcutting the installation of other packages.

Hopefully this doesn’t break when pip is upgraded by setup-python, and isn’t missing any directories that could result in partially-installed packages…

---

Edit: Per @uranusjr, added an exclusion for wheel. It’s not currently installed by Github’s setup-python action under Ubuntu, but may be in the future.

And to reiterate the bigger idea here: regardless of Github or CI, installing Python then overwriting parts of it with a previously-saved cache can break your environment. The “solution” above works today, but may not be reliable.

[ZoomIsland]

In our yaml file for Github Actions, we had cached our Python environment. Commenting out the cache options entirely fixed the issue as it re-downloaded the entirety of pip. I’m leaving this issue open, as it seems like this should still be investigated.