how to handle numpy.distutils and setuptools interaction

Hi setuptools devs, I’d like to use this issue to summarize issues in the interaction between setuptools and numpy.distutils, and then see if we can find a way to resolve those structurally.

The setuptools 50.0 release gave NumPy and SciPy (a heavy numpy.distutils users) a number of concrete problems:

• gh-2368, CCompiler_spawn() got an unexpected keyword argument 'env', also reported to NumPy at https://github.com/numpy/numpy/issues/17216
• gh-2367, AIX build of SciPy broken, also reported to SciPy at https://github.com/scipy/scipy/issues/12797
• https://github.com/numpy/numpy/pull/17209, CI break in NumPy, from distutils import sysconfig broken on TravisCI
• https://github.com/scipy/scipy/pull/12798, SciPy CI breakage due to a valid LDFLAGS fix showing up in setuptools._distutils, which gave the SciPy build a new -ffat-lto-objects that it couldn’t handle.
• There’s likely more that’s not yet reported; wheel builds of multiple packages using older NumPy versions that haven’t pinned setuptools are definitely broken. Same for CI/deployment pipelines on ARM64 and other platforms that don’t have wheels.

All of the above are resolvable. The bigger issue here is the interaction between distutils, numpy.distutils and setuptools. Due to setuptools release cadence, such breaks will keep on happening due to incompatibilities between numpy.distutils and setuptools. Let me summarize the issue, and then some options for dealing with it.

Old situation, for setuptools <50.0:

1. numpy.distutils extends and monkeypatches distutils.
2. the import order in NumPy and SciPy setup.py is:

import setuptools  # unused, just to let setuptools do its monkeypatching dance first
from numpy.distutils.core import setup

This situation worked reasonably well, because:

• distutils moved slowly
• larger distutils changes typically showed up in pre-releases of new Python versions, or in bugfix releases (say 3.7.7) that vendors like Anaconda qualify before they ship it, so we could adjust numpy.distutils to those changes
• setuptools didn’t do all that much that was relevant build-wise

New situation, for setuptools 50.0:

(default situation, no env var set)

1. setuptools replaces distutils with its vendored setuptools._distutils, even if plain distuils is imported first
2. numpy.distutils is unchanged, so it still extends and monkeypatches distutils - which is now setuptools._distutils

So numpy.distutils finds itself monkeypatching setuptools code all of sudden, and that setuptools code includes patches from Python 3.9-dev that are therefore now released into the wild with new setuptools releases without any alpha/beta/QA trajectory like we had before.

The releasing new patches quickly without testing will still be an issue for numpy.distutils probably, even if the SETUPTOOLS_USE_DISTUTILS="local" behaviour gets reverted for the time being (which seems likely, since there’s a ton of other issues right now, like the Debian breakage).

What now?

Longer-term I don’t think it’s feasible for numpy.distutils to work as it does today and extend setuptools; the release cycle mismatch will be too much of a problem. I’m not quite sure what the best solution for that is though. Options could include:

• for every NumPy release, add a hard setuptools <= current_setuptools_version constraint (NumPy releases get supported for 2.5 years by most of the scientific ecosystem though (i.e. ~50% of Python’s user base), so that’ll mean pretty old setuptools versions for building, e.g., wheels of packages that rely on NumPy - example: scikit-learn needs to build with NumPy 1.13.3 right now, so it would get setuptools 36.5)
• make numpy.distutils a standalone package on top of setuptools (then at least release cycle can be matched, but situation remains unstable)
• integrate numpy.distutils into setuptools (more testing will then be done before setuptools releases and no extending/monkeypatching is needed, but is it a good idea knowledge and work load sharing wise?)

Let me emphasize that I do see the upsides in merging disutils into setuptools, both design and maintenance wise. And we (NumPy & scientific packages) have been burned in the past by distutils patches going unmerged for ages, so setuptools being better maintained is great.

Also, for context, it may be useful to list what numpy.distutils adds besides carrying around needed distutils patches:

• Fortran build support
• SIMD intrinsics support
• BLAS/LAPACK library support (OpenBLAS, MKL, ATLAS, Netlib LAPACK/BLAS, BLIS, 64-bit ILP interface, etc.)
• Support for a few other scientific libraries, like FFTW and UMFPACK (less often used)
• Better MinGW support
• Per-compiler build flag customization (e.g. -O3 and SSE2 flags are default)
• EDIT: a simple user build config system, see site.cfg.example

Looking forward to hearing your thoughts on this topic.

[rgommers]

I see that the change in plans for NumPy hasn’t yet been posted here, so let me do so now. numpy.distutils is deprecated, and will go away for Python releases where plain distutils goes away. Users can migrate to another build system, or help add the feature(s) they need to setuptools. See https://numpy.org/devdocs/reference/distutils_status_migration.html

[rgommers]

The planned timeline is “by the time we need it for Python 3.12”, because we kinda have to. It’s still a big job though, so it depends on when we can make some dedicated time for the right person(s).

I made a start in https://github.com/rgommers/numpy/tree/meson, and the configure checks turned out to be a lot easier than with disutils. Compiler support is mostly figured out too, because that’s common with SciPy. The main sticking point will be SIMD support, see numpy.distutils.ccompiler_opt and this diagram in the docs