New version of virtualenv causes ModuleNotFoundError: No module named 'setuptools'

describe your issue

Our pre-commit CI step started failing with this error, and we can see that it now uses virtualenv 20.11.0 instead of 20.10.0 (started failing soon as they released 20.11.0). Changelog mentions:

Upgrade embedded setuptools to 59.4.0 from 58.3.0

full output:

An unexpected error has occurred: CalledProcessError: command: (‘/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/bin/python’, ‘-mpip’, ‘install’, ‘.’) return code: 2 expected return code: 0 stdout: Processing /home/runner/.cache/pre-commit/repo59pjdrh5 Installing build dependencies: started Installing build dependencies: finished with status ‘done’ Getting requirements to build wheel: started Getting requirements to build wheel: finished with status ‘done’

stderr: ERROR: Exception: Traceback (most recent call last): File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/cli/base_command.py”, line 164, in exc_logging_wrapper status = run_func(*args) File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/cli/req_command.py”, line 205, in wrapper return func(self, options, args) File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/commands/install.py”, line 338, in run requirement_set = resolver.resolve( File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py”, line 73, in resolve collected = self.factory.collect_root_requirements(root_reqs) File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py”, line 468, in collect_root_requirements req = self._make_requirement_from_install_req( File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py”, line 430, in _make_requirement_from_install_req cand = self._make_candidate_from_link( File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py”, line 201, in _make_candidate_from_link self._link_candidate_cache[link] = LinkCandidate( File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py”, line 281, in init super().init( File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py”, line 156, in init self.dist = self._prepare() File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py”, line 225, in _prepare dist = self._prepare_distribution() File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py”, line 292, in _prepare_distribution return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True) File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/operations/prepare.py”, line 482, in prepare_linked_requirement return self._prepare_linked_requirement(req, parallel_builds) File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/operations/prepare.py”, line 546, in _prepare_linked_requirement dist = _get_prepared_distribution( File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/operations/prepare.py”, line 58, in _get_prepared_distribution abstract_dist.prepare_distribution_metadata(finder, build_isolation) File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py”, line 47, in prepare_distribution_metadata self._install_build_reqs(finder) File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py”, line 106, in _install_build_reqs build_reqs = self._get_build_requires_wheel() File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py”, line 83, in _get_build_requires_wheel return backend.get_requires_for_build_wheel() File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py”, line 172, in get_requires_for_build_wheel return self._call_hook(‘get_requires_for_build_wheel’, { File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py”, line 332, in _call_hook raise BackendUnavailable(data.get(‘traceback’, ‘’)) pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last): File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py”, line 89, in _build_backend obj = import_module(mod_path) File “/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/importlib/init.py”, line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File “<frozen importlib._bootstrap>”, line 1030, in _gcd_import File “<frozen importlib._bootstrap>”, line 1007, in _find_and_load File “<frozen importlib._bootstrap>”, line 986, in _find_and_load_unlocked File “<frozen importlib._bootstrap>”, line 680, in _load_unlocked File “<frozen importlib._bootstrap_external>”, line 850, in exec_module File “<frozen importlib._bootstrap>”, line 228, in _call_with_frames_removed File “/tmp/pip-build-env-rf7ihm0k/overlay/lib/python3.9/site-packages/poetry/masonry/api.py”, line 1, in <module> from poetry.core.masonry.api import build_sdist File “/tmp/pip-build-env-rf7ihm0k/overlay/lib/python3.9/site-packages/poetry/core/masonry/init.py”, line 10, in <module> from .builder import Builder File “/tmp/pip-build-env-rf7ihm0k/overlay/lib/python3.9/site-packages/poetry/core/masonry/builder.py”, line 7, in <module> from .builders.sdist import SdistBuilder File “/tmp/pip-build-env-rf7ihm0k/overlay/lib/python3.9/site-packages/poetry/core/masonry/builders/init.py”, line 2, in <module> from .wheel import WheelBuilder File “/tmp/pip-build-env-rf7ihm0k/overlay/lib/python3.9/site-packages/poetry/core/masonry/builders/wheel.py”, line 23, in <module> from packaging.tags import sys_tags File “/tmp/pip-build-env-rf7ihm0k/overlay/lib/python3.9/site-packages/poetry/core/_vendor/packaging/tags.py”, line 7, in <module> import distutils.util File “/home/runner/.cache/pre-commit/repo59pjdrh5/py_env-python3.9/lib/python3.9/site-packages/_distutils_hack/init.py”, line 92, in create_module return importlib.import_module(‘setuptools._distutils’) File “/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/importlib/init.py”, line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ModuleNotFoundError: No module named ‘setuptools’

pre-commit --version

2.9.2

.pre-commit-config.yaml

we use:
- black
- isort
- flake8
- eslint
- debug linter
- mypy
- semgrep

~/.cache/pre-commit/pre-commit.log (if present)

No response

[asottile]

here’s the relevant things to follow up on:

• https://github.com/pypa/setuptools/issues/2353
• https://github.com/pypa/pip/issues/6264
• https://github.com/pypa/setuptools/issues/2980

why we’re suddenly seeing this:

• the latest virtualenv release upgraded setuptools to 60.1.0 (despite what the changelog says it was upgraded to 60.1.0 here)
• setuptools 60.* changes the default to using the setuptools-embedded distutils rather than the stdlib distutils
• setuptools does this by way of a .pth file which redirects imports of distutils to setuptools._distutils
• during pip’s isolated builds (triggered by pyproject.toml, for example to install isort via poetry) pip attempts to clear the current sys.path of the enclosing environment and isolates to the pyproject.toml-installed build dependencies, but it’s a bit too late as the .pth files from the enclosing environment are applied. so in this environment setuptools is not installed, but its import hooks are

workarounds:

• ~avoid installing virtualenv==20.11.0~ (will begin failing soon due to periodic update)
• set SETUPTOOLS_USE_DISTUTILS=stdlib to avoid the local default

I don’t have any say on the upstream projects, but my recommendation would be to:

• yank virtualenv==20.11.0 to avoid automatic unpinned installation of that
• fix pip to have proper build isolation against the parent site – something like this hack or use actual virtual environments to perform isolated builds
• (optional) until pip is released with a fix, revert back to =stdlib as the default in setuptools

reproducing this outside of pre-commit / without latest virtualenv / without virtualenv

python3 -m venv venv
venv/bin/pip install setuptools==60.1.0 pip==21.3.1
venv/bin/pip install --no-binary :all: isort==5.10.1

[pradyunsg]

pip 60.1.0

*setuptools