Reconsidering distutils replacement strategy

I first expressed concerns about the fact that distutils needs to be imported after setuptools in this post, and I did not realize that we had not adequately addressed this before the distutils adoption. I think we should re-consider the way this is done, to avoid any sort of requirement on the sort order.

Currently, when you import setuptools, distutils is injected into sys.modules.

The problem here is that it requires importing setuptools before distutils, which most people will not be doing right now (by almost all common import sort orders, distutils comes first).

@jaraco explains why it’s done this way here:

I did consider another approach, where Setuptools could add a .pth file that would import setuptools.distutils_patch. Such an approach would always happen earlier and so would not be subject to the race that’s happening here, but it would also happen whether or not setuptools was imported (on any invocation of Python in that environment).

I think one preferred solution would be for us to install a distutils package that would be imported before distutils. The problem is that in the standard sys.path, site-packages comes after the standard library. Since anything we do here will be a tremendous hack, I’m thinking that maybe the most surgical way to accomplish this is to do something like this:

1. Create a setuptools._distutils/import_hack/distutils/__init__.py (where there’s no __init__.py in import_hack) that invokes import setuptools.distutils_patch.
2. Add a .pth file that injects $SITEPACKAGES/setuptools/_import_hack/ into sys.path before the standard library.

Another option I think we should consider is stepping up the timetable on providing everything distutils provides directly from the setuptools namespace, as I described here:

I personally would prefer it if we made the setuptools namespace the canonical namespace, and have distutils just import symbols from setuptools and possibly wrap them in deprecation warnings.

Right now, there are things that you must import from distutils, and so we can’t say, “You can avoid an issue with the sort order by just not importing distutils”. If we provide everything that distutils provides in the setuptools namespace, we can just say, “Just use the setuptools version and you won’t have this problem.”

Personally, I think we should do a combination of both — make it so importing distutils still works in whatever order you do it in and move everything to the canonical setuptools namespace. That will make it easier to actively deprecate direct use of distutils, and pave the way for us eventually removing the need for the .pth file (though the fastest road to not having the .pth file will be via removing distutils from the standard library).

[gaborbernat]

I’d like 49.3 to prove stable then switch the default (so SETUPTOOLS_USE_DISTUTILS=stdlib is needed to opt-out instead) in about a week.

This is getting really frustrating. It’s now the third setutpools release that breaks multiple CIs of mine only because some libraries dare to use distutils LooseVersion and setuptools start spitting out a warning about what no one can do nothing about. Sigh. I really wish we’d stop with this warning craze, and move to something better. End users are the one to see it and suffer with it the most, but they can’t address it… only the library maintainers can.

Also, please don’t close an issue until it’s not fixed. This clearly isn’t, not at least the default switch is the other way around.

[mattip]

It seems you released setuptools 49.2.1 without resolving this issue (also #2260). Was that on purpose? This is causing no end of problems with downstream projects, and will slow the adoption of new versions of setuptools.