[BUG] new release (57) breaks piptools pytest

First things first: please let me know if there is any information missing - It’s kind of hard to me to trace this down to the actual bug or to an minimum example of what is failing unfortunately.

setuptools version

setuptools==57.0.0

Python version

3.6,3.7,3.8,3.9

OS

Ubuntu, Windows, MacOS

Additional environment information

Hey, please see this issue where it was discovered: https://github.com/jazzband/pip-tools/issues/1403

First, it was wrongly attributed to be caused by pip version deprecation and feature flag, but it seems to be due to new setuptools (details in description). So currently all CI runners are broken there in that project.

• Last successful run: https://github.com/jazzband/pip-tools/actions/runs/870122483
• First failing: https://github.com/jazzband/pip-tools/actions/runs/873544759
• run which was pinned to setuptools<57 and ran through: https://github.com/jazzband/pip-tools/actions/runs/885789425

Description

Unfortunately, I’m not exactly aware of what happens behind the scenes on piptools testing suite. I’d like to share the code I could reproduce locally so far by running some of the tests manually in the fields below.

Expected behavior

There shouldn’t be anything failing because it worked on the old version before.

How to Reproduce

1. in piptools package open tests/test_cli_compile.py and copy the imports from above and the code from the function test_realistic_complex_sub_dependencies, done in the following in detail:
2. navigate to tests folder and run the following:

import os
import subprocess
import sys
from textwrap import dedent
from unittest import mock
import pytest
from pip._internal.utils.urls import path_to_url
from piptools.scripts.compile import cli
from constants import MINIMAL_WHEELS_PATH, PACKAGES_PATH
from click.testing import CliRunner

1. create runner = CliRunner(mix_stderr=False)
2. run

    wheels_dir = "wheels"
    # make a temporary wheel of a fake package
    subprocess.run(
        [
            "pip",
            "wheel",
            "--no-deps",
            "-w",
            wheels_dir,
            os.path.join(PACKAGES_PATH, "fake_with_deps", "."),
        ],
        check=True,
    )
    with open("requirements.in", "w") as req_in:
        req_in.write("fake_with_deps")  # require fake package

    out = runner.invoke(cli, ["-n", "--rebuild", "-f", wheels_dir])

which fails.

Output

expected

In [39]: out.stderr
Out[39]: '#\n# This file is autogenerated by pip-compile\n# To update, run:\n#\n#    pip-compile --find-links=wheels\n#\n--find-links wheels\n\ncolorama==0.3.9\n    # via fake-with-deps\ncornice==1.0.0\n    # via fake-with-deps\nenum34==1.1.6\n    # via fake-with-deps\nfake-with-deps==0.1\n    # via -r requirements.in\nfutures==3.0.5\n    # via fake-with-deps\ngreenlet==1.1.0\n    # via sqlalchemy\nipaddress==1.0.23\n    # via fake-with-deps\njsonschema==2.6.0\n    # via fake-with-deps\npastedeploy==2.1.1\n    # via pyramid\npyramid==1.5.8\n    # via\n    #   cornice\n    #   fake-with-deps\npython-dateutil==2.4.2\n    # via fake-with-deps\npython-memcached==1.59\n    # via fake-with-deps\npyzmq==14.7.0\n    # via fake-with-deps\nrepoze.lru==0.7\n    # via pyramid\nsimplejson==3.17.2\n    # via\n    #   cornice\n    #   fake-with-deps\nsix==1.8.0\n    # via\n    #   fake-with-deps\n    #   python-dateutil\n    #   python-memcached\nsqlalchemy==1.4.15\n    # via fake-with-deps\ntranslationstring==1.4\n    # via pyramid\nvenusian==3.0.0\n    # via pyramid\nwebob==1.8.7\n    # via pyramid\nxmltodict==0.11.0\n    # via fake-with-deps\nzope.deprecation==4.4.0\n    # via pyramid\nzope.interface==5.4.0\n    # via pyramid\n\n# The following packages are considered to be unsafe in a requirements file:\n# setuptools\nDry-run, so nothing updated.\n'

actual (failing) output

      ERROR: Command errored out with exit status 1:
       command: /Users/nalbers/git/pip-tools/thisvirtualenv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"'; __file__='"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-pip-egg-info-nkzz3eb8
           cwd: /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/
      Complete output (29 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/__init__.py", line 18, in <module>
          from setuptools.dist import Distribution
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/_virtualenv.py", line 89, in exec_module
          old(module)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/dist.py", line 32, in <module>
          from setuptools.extern.more_itertools import unique_everseen
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
        File "<frozen importlib._bootstrap>", line 565, in module_from_spec
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 52, in create_module
          return self.load_module(spec.name)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 37, in load_module
          __import__(extant)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
          from .more import *  # noqa
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
          from concurrent.futures import ThreadPoolExecutor
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/__init__.py", line 8, in <module>
          from concurrent.futures._base import (FIRST_COMPLETED,
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/_base.py", line 357
          raise type(self._exception), self._exception, self._traceback
                                     ^
      SyntaxError: invalid syntax
      ----------------------------------------

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

Indeed, I can replicate the issue without pip thus:

draft $ http https://files.pythonhosted.org/packages/25/83/113da7c7832a4efa80ec7fd5d25b2ea431d9a847cd78f1fa98ddd99d591a/futures-3.1.0.tar.gz | tar xz
draft $ cd futures-3.1.0/
futures-3.1.0 $ pip-run -q setuptools -- setup.py egg_info
/Users/jaraco/draft/futures-3.1.0/setup.py:6: UserWarning: This backport is meant only for Python 2.
Python 3 users do not need it, as the concurrent.futures package is available in the standard library.
  warn('This backport is meant only for Python 2.\n'
Traceback (most recent call last):
  File "/Users/jaraco/draft/futures-3.1.0/setup.py", line 12, in <module>
    from setuptools import setup
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/dist.py", line 32, in <module>
    from setuptools.extern.more_itertools import unique_everseen
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/extern/__init__.py", line 52, in create_module
    return self.load_module(spec.name)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/extern/__init__.py", line 37, in load_module
    __import__(extant)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
    from .more import *  # noqa
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ug2uyft1/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
    from concurrent.futures import ThreadPoolExecutor
  File "/Users/jaraco/draft/futures-3.1.0/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/Users/jaraco/draft/futures-3.1.0/concurrent/futures/_base.py", line 357
    raise type(self._exception), self._exception, self._traceback
                               ^
SyntaxError: invalid syntax
futures-3.1.0 $ python -c "import concurrent; print(concurrent.__file__)"
/Users/jaraco/draft/futures-3.1.0/concurrent/__init__.py

[PatTheMav]

Yep exactly, it’s the perfect storm of setuptools using a dependency that imports concurrent.futures since v57 and root imports having higher priority than stdlib imports (which “protected” Py3 against this incompatible package so far).

Only installing the futures package itself could lead to this issue, as it’s the only one containing an actual concurrent.futures module (or more precisely, any package that has such a module with code that isn’t Py3 compatible) available at the project root.