[BUG] Having setuptools installed causes cpython stdlib build to fail

setuptools version

setuptools>=60

Python version

Python 3.9.9 (and probably others)

OS

Probably affects all; reported on macOS

Additional environment information

Originally reported in the context of MacPorts’ python39 and py39-setuptools ports.

Description

If you have setuptools installed and attempt to build cpython from source, setuptools’ copy of distutils may be used when building the stdlib modules and cause the build to fail due to an incompatibility.

This is easily cured by setting SETUPTOOLS_USE_DISTUTILS=stdlib if you understand the problem, but if you don’t, it’s a pretty baffling failure mode.

Downstream report: https://trac.macports.org/ticket/64352

Expected behavior

Cpython build uses its own distutils and succeeds.

How to Reproduce

1. Install setuptools into a convenient site-packages directory.
2. Configure cpython to install in a prefix such that it will be using the site-packages directory from step 1.
3. Build cpython.

Output

Traceback (most recent call last):
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 2598, in <module>
    main()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 2568, in main
    setup(# PyPI Metadata (PEP 301)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 456, in build_extensions
    build_ext.build_extensions(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 446, in build_extensions
    self._build_extensions_parallel()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in _build_extensions_parallel
    fut.result()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/Lib/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/Lib/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/Lib/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9/./setup.py", line 554, in build_extension
    build_ext.build_extension(self, ext)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 117, in _compile
    self.spawn(compiler_so + cc_args + [src, '-o', obj] +
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 917, in spawn
    spawn(cmd, dry_run=self.dry_run, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_distutils/spawn.py", line 38, in spawn
    log.info(subprocess.list2cmdline(cmd))
AttributeError: module '_bootsubprocess' has no attribute 'list2cmdline'
make: *** [sharedmods] Error 1
make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9'
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python39/python39/work/Python-3.9.9" && /usr/bin/make -j2 -w all

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

I’m a bit confused as to whether the issue is actually fixed or not, though.

The issue reported here was addressed (stdlib builds do work for the most part). It appears, however, that there are some select cases where the CPython build detection isn’t taking effect, so continues to lead to trouble.

Let’s continue the conversation in python/cpython#91169, which is the open issue describing the defect.

[nre-ableton]

I ran into this behavior recently while trying to build Python 3.10.4 on Ubuntu 22.04, which resulted in a confusing error, ModuleNotFoundError: No module named 'binascii'. Thankfully exporting SETUPTOOLS_USE_DISTUTILS=stdlib worked, but I agree it’s not an ideal workaround.

This issue was closed, having been fixed by #3031. I’m a bit confused as to whether the issue is actually fixed or not, though.