Expose compiler classes for customized use

Summary

The porting from distutils docs are a bit sparse, and could use some more specific examples. Specifically, for distutils.ccompiler and distutils.util.get_platform were the replacements I was looking for, and didn’t find.

Maybe it would be good to start populating a FAQ section at or linked from the docs. Starting out with finishing the the table in the pep with actual recommended import substitutions that work (4 are there already), would be a huge help for migrators like myself.

OS / Environment

No response

Additional Information

In attempting to migrate https://github.com/zeromq/pyzmq away from distutils, I came up with some questions that don’t appear to have clear answers in docs, and it seems like setuptools is the right place for such docs (if that’s wrong, what is the right place?).

The pep lists setuptools as the thing to switch to for several APIs, such as ccompiler, etc., but at least ccompiler does not appear to be available from public setuptools APIs. Is importing form setuptools._distutils the ‘right’ way in general to get distutils functionality that hasn’t been exposed at public level of setuptools yet? Or are there plans to expose these as top-level APIs (e.g. setutools.ccompiler)?

I can import new_compiler and customize_compiler from setuptools.command.build_ext, but that doesn’t seem better than setuptools._distutils.ccompiler to me. If it were documented as the suggested alternative, I’d be happy, though.

Could we perhaps have a statement in the docs about when, if ever, importing from setuptools._distutils is appropriate? The text already there:

If a project relies on uses of distutils that do not have a suitable replacement above, please search the Setuptools issue tracker and file a request

suggests anything that requires importing from setuptools._distutils should result in an Issue (how I got here), but maybe could be more explicit about what to do while finding a resolution. A blessing of “import from setuptools._distutils” in the interim would ease some stress on migrators, but may not be what you all want to support.

The pep also lists the platform module as the replacement for distutils.util.get_platform, but in my experience this is not the right advice. Where I’ve used distutils.util.get_platform(), what I really wanted was the build platform string, which is available as dist.get_command_obj('build').plat_name. I don’t always call it from within a command so this is a bit inconvenient, but I think I can manage. I don’t think it’s possible to construct this from the platform module.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

Does it mean that distutils can be imported even in Python 3.12 if setuptools is installed ?

Yes. That’s correct. The only exception is that if SETUPTOOLS_USE_DISTUTILS=stdlib is set, then no distutils will be found. However, the default since Setuptools 60 is SETUPTOOLS_USE_DISTUTILS=local, so that default behavior remains unchanged across Python versions, including 3.12+.

if you guys plan to maintain this interface

We do, though it’s our goal to obviate the need for accessing distutils altogether (eventually deprecating and removing it). Meaning that we do want to identify use-cases like the ones described above and provide a robust, long-term solution that doesn’t involve distutils. That’s why I’d like to avoid simply exposing distutils interfaces in Setuptools and instead step back and think about the problems that these approaches address. I need to create a milestone to track these efforts.

It seems to me from the responses above that there are basically these use-cases we want to meet:

• Detect and expose platform-specific configuration and capabilities relating to the compiler.
• Set platform-specific flags before building extensions (under Setuptools).
• Compile shared libraries to be included in a distribution.
• [enhancement] Provide a customized compiler to the build process (for experimentation or customization).

I’m thinking we should create new issues for each and close this issue as “invalid” as the compatibility shim already exposes the compiler classes. Does that sound right?

[althonos]

@jaraco : Thanks for tackling this issue!

Can you help clarify the use-case for these classes/functions, i.e. under what conditions do you need to import a CCompiler? Under what conditions do you call new_compiler? What do you do with the results? Do you do this exclusively in service of building a package (alongside other Setuptools operations) or is it something your project does independently?

I have the same use-case in several scientific computing packages, where I have several components:

• a C or C++ library that I’m writing bindings for
• a Cython extension exposing a Python interface
• several additional files which are platform-specific (e.g. a specialized SSE2 implementation of some algorithms).

In general, I need to have access to a C compiler before building the Cython extension so that I can detect some platform-specific functions and headers (like a configure script would), and also to check if I can build the specialized code (i.e. does the compiler support SSE2/AVX/NEON etc.). I have a (non-minimal) example here: https://github.com/althonos/pytrimal/blob/main/setup.py

At the moment I’m never creating a CCompiler directly, I’m actually extending the build_ext command so that I can use self.compiler, which is a CCompiler configured with customize_compiler. But if a dedicated compilers package was available, I’d likely use that instead 👍