Ability to force distutils as installation scheme backend even on Python 3.10

What’s the problem this feature will solve?

Hello, as you might already know from discussions in https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/120, in Fedora Linux, we patch our Python to install packages to /usr/local prefix by default. This has been historically done by patching distutils.command.install and recently changed to patching sysconfig._INSTALL_SCHEMES.

I couldn’t be happier with pip’s move towards the sysconfig way, as it makes everything nicer at the end and allows us to get rid of distutils once for all, as distutils is riddled with traps.

I see that pip made a considerable choice here and only changes the behavior in Python 3.10+. However, this puts us (the Fedora’s Python maintainers) into an unfortunate situation:

• we have updated our “main” Python version to 3.10 in Fedora Linux 35
• we have only changed our patch to patch sysconfig in Fedora Linux 36+

We realized along the way that the change is not 100% backward compatible and it uncovered several weird bugs in various programs, for example:

• Our package manager failed to see its own plugins.
• Our distro installer failed to properly get a list of Python files to put into the initial memory image for the bootloader.

Unfortunately, when pip 21.3 was released, Fedora Linux 35 was already beyond code freeze and stability guarantees apply now. As much as I would like to do this change in Fedora Linux 35 as well, I have not found a way to do it that’s not potentially disturbing to our users.

What happens now is that when users of Fedora Linux 35 use pip to upgrade pip, it’ll start installing packages back to the /usr prefix, which has the potential to brick their systems. We don’t have any data, but my assumption is that users who pip install --upgrade pip are more likely to actually sudo pip install something that’ll negatively impact the package manager or a similar tool.

pip-installed pip 21.2.3:

[root@dde39f841e21 /]# pip install pello
Collecting pello
  Using cached Pello-1.0.2-py3-none-any.whl (6.0 kB)
Installing collected packages: pello
Successfully installed pello-1.0.2

[root@dde39f841e21 /]# pip uninstall pello
Found existing installation: Pello 1.0.2
Uninstalling Pello-1.0.2:
  Would remove:
    /usr/local/bin/pello_greeting
    /usr/local/lib/python3.10/site-packages/Pello-1.0.2.dist-info/*
    /usr/local/lib/python3.10/site-packages/pello/*
Proceed (Y/n)?

pip-installed pip 21.3:

[root@dde39f841e21 /]# pip install pello
Collecting pello
  Downloading Pello-1.0.2-py3-none-any.whl (6.0 kB)
Installing collected packages: pello
Successfully installed pello-1.0.2

[root@dde39f841e21 /]# pip uninstall pello
Found existing installation: Pello 1.0.2
Uninstalling Pello-1.0.2:
  Would remove:
    /usr/bin/pello_greeting
    /usr/lib/python3.10/site-packages/Pello-1.0.2.dist-info/*
    /usr/lib/python3.10/site-packages/pello/*
Proceed (Y/n)?

This does not happen for Fedora Linux 33 and Fedora Linux 34, as the “main” Python version si 3.9 there. This does not happen to Fedora Linux 36 users, as we patch sysconfig and hence our setup is compatible with pip’s decision.

Now, I am not asking you to revert this decision. I am not asking to postpone it to Python 3.11 either. I would merely like to have a way for the Python interpeter to tell pip “hey, I am not ready for this just yet, please use distutils”. We would then be able to do this.

Note that we cannot “simply” patch the pip we ship, because it’s the upstream pip that the users will likely to install.

Describe the solution you’d like

In here:

https://github.com/pypa/pip/blob/21.3.1/src/pip/_internal/locations/__init__.py#L48

I’d like to be able to able to inject an additional condition that would allow us to use distutils for a bit more time. Something like this:

_USE_SYSCONFIG = getattr(sysconfig, '_PIP_USE_SYSCONFIG', sys.version_info >= (3, 10))

That way, we could set sysconfig._PIP_USE_SYSCONFIG = False in Python 3.10 on Fedora 35. We could also set sysconfig._PIP_USE_SYSCONFIG = True in Python 3.9 on Fedora 36+ if we would like to have that experience unified across our interpreters.

Alternative Solutions

EDIT: See an alternate solution in https://github.com/pypa/pip/issues/10647#issuecomment-964261189

An alternate solution is to document this more heavily and discourage users to use sudo pip, but we already do this and it doesn’t work 😦

Additional context

Fedora’s downstream issue: https://bugzilla.redhat.com/show_bug.cgi?id=2014513

Code of Conduct

• I agree to follow the PSF Code of Conduct.

Thank you for considering this. I am willing to add the code if we agree on the behavior.

[uranusjr]

I see, that makes sense. A private attribute is best then.

[hroncok]

I am preparing a Python build for Fedora 35 that sets this attribute and will test if it works as expected.

Will submit a draft PR shorty, and later figure out the best approach to make this tested.