setuptools>=42 encounters TypeError when installing a package with an extension that was built with setuptools<42.

Description of the problem

There is a breaking change in setuptools 42 that prevents it (and any newer version) from consuming certain packages built with older version of setuptools. If the package contains a Cython extension and is built using setuptools 41.6.0 or older, then setuptools 42 or newer will trigger a TypeError when installing the package. It looks like an internal object is not being fully initialized for some reason.

Steps to reproduce

I’ve attached archives of a minimal test library and minimal application that shows the problem: application.tar.gz library.tar.gz

First build the library:

mkdir library
tar -C library -xzf library.tar.gz
pushd library

python3 -m venv .venv --prompt "$(basename $(pwd))"
. .venv/bin/activate
  python setup.py sdist
deactivate
popd

Copy/upload library/dist/test-library-0.0.0.tar.gz to a repo that can be accessed later when we set up the application.

Now try to set up the virtual environment for the application:

mkdir application
tar -C application -xzf application.tar.gz
pushd application

python3 -m venv .venv --prompt "$(basename $(pwd))"
. .venv/bin/activate
  pip install 'wheel==0.34.2' 'setuptools==42.0.0'  # Can be any newer version of setuptools as well. E.g., 49.2.1
  pip install --no-cache-dir -e .                   # <-- problem is here
deactivate
popd

When running the last pip install step, I get this error:

Building wheels for collected packages: test-library
  Building wheel for test-library (setup.py): started
  Building wheel for test-library (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /tmp/setuptools-bug-mcve/application/.venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fh10g_zv/test-library/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fh10g_zv/test-library/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-1g5vplsd
       cwd: /tmp/pip-install-fh10g_zv/test-library/
  Complete output (44 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/somepkg
  copying somepkg/__init__.py -> build/lib.linux-x86_64-3.6/somepkg
  running egg_info
  writing test_library.egg-info/PKG-INFO
  writing dependency_links to test_library.egg-info/dependency_links.txt
  writing top-level names to test_library.egg-info/top_level.txt
  reading manifest file 'test_library.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'test_library.egg-info/SOURCES.txt'
  copying somepkg/hello.pyx -> build/lib.linux-x86_64-3.6/somepkg
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-fh10g_zv/test-library/setup.py", line 17, in <module>
      Extension('somepkg.hello', sources=[os.path.join('somepkg', 'hello.pyx')])
    File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/setuptools/command/build_py.py", line 53, in run
      self.build_package_data()
    File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/setuptools/command/build_py.py", line 126, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  TypeError: argument of type 'NoneType' is not iterable
  ----------------------------------------
  ERROR: Failed building wheel for test-library
  Running setup.py clean for test-library
Failed to build test-library
DEPRECATION: Could not build wheels for test-library which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Installing collected packages: test-library, test-application
    Running setup.py install for test-library: started
    Running setup.py install for test-library: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /tmp/setuptools-bug-mcve/application/.venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fh10g_zv/test-library/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fh10g_zv/test-library/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0r3n4uiw/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/setuptools-bug-mcve/application/.venv/include/site/python3.6/test-library
         cwd: /tmp/pip-install-fh10g_zv/test-library/
    Complete output (46 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/somepkg
    copying somepkg/__init__.py -> build/lib.linux-x86_64-3.6/somepkg
    running egg_info
    writing test_library.egg-info/PKG-INFO
    writing dependency_links to test_library.egg-info/dependency_links.txt
    writing top-level names to test_library.egg-info/top_level.txt
    reading manifest file 'test_library.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'test_library.egg-info/SOURCES.txt'
    copying somepkg/hello.pyx -> build/lib.linux-x86_64-3.6/somepkg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-fh10g_zv/test-library/setup.py", line 17, in <module>
        Extension('somepkg.hello', sources=[os.path.join('somepkg', 'hello.pyx')])
      File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/opt/rh/rh-python36/root/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/setuptools/command/build_py.py", line 53, in run
        self.build_package_data()
      File "/tmp/setuptools-bug-mcve/application/.venv/lib64/python3.6/site-packages/setuptools/command/build_py.py", line 126, in build_package_data
        srcfile in self.distribution.convert_2to3_doctests):
    TypeError: argument of type 'NoneType' is not iterable
    ----------------------------------------
ERROR: Command errored out with exit status 1: /tmp/setuptools-bug-mcve/application/.venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fh10g_zv/test-library/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fh10g_zv/test-library/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0r3n4uiw/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/setuptools-bug-mcve/application/.venv/include/site/python3.6/test-library Check the logs for full command output.

This is as reported by pip==20.2.1, but the same error (albeit less verbose) occurs with versions as old as pip==9.0.1.

In case it is pertinent, this is running with Python 3.6 on CentOS 7.

[kwvanderlinde]

@jaraco I’ve gone and removed setuptools from setup_requires and am not seeing any further issues on our end. It turns out I don’t need a specific setuptools version when installing the package, just when building it, so there is no reason for us to declare this dependency.

[jaraco]

I found the issue - it’s the presence of setup_requires=['setuptools']. Removing that allows the project to build properly:

draft $ mkdir library; http --follow https://github.com/pypa/setuptools/files/5043273/library.tar.gz | tar xz -C library; pushd library
~/draft/library ~/draft
library $ sed -ie "/setuptools ==/d" setup.py
library $ pip-run -q setuptools==49.3 -- setup.py build
WARNING: The wheel package is not available.
running build
running build_py
creating build
creating build/lib.macosx-10.9-x86_64-3.8
creating build/lib.macosx-10.9-x86_64-3.8/somepkg
copying somepkg/__init__.py -> build/lib.macosx-10.9-x86_64-3.8/somepkg
running egg_info
creating test_library.egg-info
writing test_library.egg-info/PKG-INFO
writing dependency_links to test_library.egg-info/dependency_links.txt
writing top-level names to test_library.egg-info/top_level.txt
writing manifest file 'test_library.egg-info/SOURCES.txt'
reading manifest file 'test_library.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'test_library.egg-info/SOURCES.txt'
copying somepkg/hello.pyx -> build/lib.macosx-10.9-x86_64-3.8/somepkg
running build_ext
cythoning somepkg/hello.pyx to somepkg/hello.c
/Users/jaraco/draft/library/.eggs/Cython-0.29.21-py3.8-macosx-10.9-x86_64.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /Users/jaraco/draft/library/somepkg/hello.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
building 'somepkg.hello' extension
creating build/temp.macosx-10.9-x86_64-3.8
creating build/temp.macosx-10.9-x86_64-3.8/somepkg
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c somepkg/hello.c -o build/temp.macosx-10.9-x86_64-3.8/somepkg/hello.o
gcc -bundle -undefined dynamic_lookup -arch x86_64 -g build/temp.macosx-10.9-x86_64-3.8/somepkg/hello.o -o build/lib.macosx-10.9-x86_64-3.8/somepkg/hello.cpython-38-darwin.so

Here’s what I think’s happening - the presence of setup_requires with a pinned setuptools version means that setuptools will attempt to download and install that specific version of setuptools if it’s not already present. Moreover, it will attempt to inject that version into setuptools into the working set at run time, leading to undefined behaviors (such as metadata like entry points being missing).

Is the easy fix here to simply remove the setuptools declaration there?