[BUG] On Debian, install fails with "no such option 'install_layout'".

setuptools version

60.0.4

Python version

Python 3.9.9

OS

Ubuntu 21.10

Additional environment information

Running DEB packaging with dh-python.

Description

In Debian setuptools packaging an additional patch applied: https://sources.debian.org/src/python-setuptools/44.1.1-1/debian/patches/install-layout.diff

During 59.8.0, running debuild -uc -us for building setuptools WITHOUT about patch keep functioning.

During 60.0.0+, no matter with / without about patch, running debuild -uc -us keep failing.

If modify the build script as below, build now OK:

 override_dh_auto_install:
-       dh_auto_install
+       export SETUPTOOLS_USE_DISTUTILS=stdlib && dh_auto_install

Expected behavior

For v59.8.0 WITHOUT any additional patch it is OK:

sudo podman run -ti --rm ubuntu:21.10 bash
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v59.8.0 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
git clean -xdf
tar zcvf ../python-setuptools_59.8.0.orig.tar.gz --exclude=.git .
debuild -uc -us

For v60.0.4 WITH SETUPTOOLS_USE_DISTUTILS=stdlib it is OK:

sudo podman run -ti --rm ubuntu:21.10 bash
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
git clean -xdf
tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .
debuild -uc -us

How to Reproduce

For v60.0.4 WITHOUT SETUPTOOLS_USE_DISTUTILS=stdlib it is FAILED:

sudo podman run -ti --rm ubuntu:21.10 bash
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
find *.spec debian/rules -type f | xargs sed -i 's/export SETUPTOOLS_USE_DISTUTILS=stdlib && //g'
git clean -xdf
tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .
debuild -uc -us

Output

   debian/rules override_dh_auto_install
make[1]: Entering directory '/pypa-setuptools'
dh_auto_install
I: pybuild base:232: /usr/bin/python3 setup.py install --root /pypa-setuptools/debian/tmp 
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'build-lib' will not be supported in future versions. Please use the underscore name 'build_lib' instead
  warnings.warn(
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-layout' will not be supported in future versions. Please use the underscore name 'install_layout' instead
  warnings.warn(
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-scripts' will not be supported in future versions. Please use the underscore name 'install_scripts' instead
  warnings.warn(
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-lib' will not be supported in future versions. Please use the underscore name 'install_lib' instead
  warnings.warn(
running install
/pypa-setuptools/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
error: error in /pypa-setuptools/.pybuild/cpython3_3.9/.pydistutils.cfg: command 'install_with_pth' has no such option 'install_layout'
E: pybuild pybuild:353: install: plugin distutils failed with: exit code=1: /usr/bin/python3 setup.py install --root /pypa-setuptools/debian/tmp 
dh_auto_install: error: pybuild --install -i python{version} -p 3.9 --dest-dir /pypa-setuptools/debian/tmp returned exit code 13
make[1]: *** [debian/rules:6: override_dh_auto_install] Error 25
make[1]: Leaving directory '/pypa-setuptools'
make: *** [debian/rules:16: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui failed

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

In pypa/distutils@69f85733543b9f6257e89c4f79f9507f20d06b49 (debian-patch branch), I’ve added one additional step that extends the install schemes in sysconfig to include the deb_system. And now the repro above builds without error:

$ cat Dockerfile
FROM ubuntu:21.10
RUN apt update
RUN apt -y full-upgrade
RUN DEBIAN_FRONTEND=noninteractive apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
RUN git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git
WORKDIR pypa-setuptools
RUN find *.spec debian/rules -type f | xargs sed -i 's/export SETUPTOOLS_USE_DISTUTILS=stdlib && //g'
RUN curl -skL https://raw.githubusercontent.com/pypa/distutils/69f8573354/_distutils_system_mod.py > /usr/lib/python3/dist-packages/_distutils_system_mod.py
RUN git clean -xdf
RUN tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .
RUN debuild -uc -us
$ 
$ docker build .
[+] Building 73.5s (15/15) FINISHED                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                      0.0s
 => => transferring dockerfile: 727B                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                         0.0s
 => => transferring context: 2B                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/ubuntu:21.10                                                                           0.3s
 => [ 1/11] FROM docker.io/library/ubuntu:21.10@sha256:cc8f713078bfddfe9ace41e29eb73298f52b2c958ccacd1b376b9378e20906ef                   0.0s
 => CACHED [ 2/11] RUN apt update                                                                                                         0.0s
 => CACHED [ 3/11] RUN apt -y full-upgrade                                                                                                0.0s
 => [ 4/11] RUN DEBIAN_FRONTEND=noninteractive apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-esse  47.6s
 => [ 5/11] RUN git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git                                           6.8s 
 => [ 6/11] WORKDIR pypa-setuptools                                                                                                       0.0s 
 => [ 7/11] RUN find *.spec debian/rules -type f | xargs sed -i 's/export SETUPTOOLS_USE_DISTUTILS=stdlib && //g'                         0.2s 
 => [ 8/11] RUN curl -skL https://raw.githubusercontent.com/pypa/distutils/69f8573354/_distutils_system_mod.py > /usr/lib/python3/dist-p  0.6s 
 => [ 9/11] RUN git clean -xdf                                                                                                            0.2s 
 => [10/11] RUN tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .                                                         0.6s 
 => [11/11] RUN debuild -uc -us                                                                                                          10.8s
 => exporting to image                                                                                                                    6.2s 
 => => exporting layers                                                                                                                   6.2s 
 => => writing image sha256:e01c723814752a10875080257043a0e10e94076418d1f91c21ac34501d639cf1                                              0.0s 

Is that sufficient to close this issue, at least until Debian maintainers step up with an alternative proposal?

[jaraco]

4. Support --install-layout officially inside setuptools

I considered this approach, but there are two problems with the approach:

1. The design is Debian-specific and applies Debian-specific behaviors. It was designed with a very narrow scope and doesn’t consider other platforms. If the feature can be shown to have general-purpose benefit and be submitted as a PR to this project, that may be acceptable.
2. The design runs in conflict with the general-purpose approach, which has been coordinated with other platforms and which is available to other installer tools (flit, build, pip) via sysconfig (install schemes and get_preferred_scheme). It’s imperative that Debian utilize the general-purpose approach and if it’s inadequate, describe what the gaps are and propose a solution that fills those gaps.

2. Provide a global on/off switch DURING setuptools installation, e.g.

Remember that SETUPTOOLS_USE_DISTUTILS=stdlib is a temporary escape hatch. It’s there to enable end users and installers to opt out of the inevitable transition to a world without distutils in the stdlib (slated for Python 3.12). Making this change permanent in an environment does simplify orchestration of that environment, but it also has the following downsides:

• Users can no longer experiment with installing their packages with =local.
• The platform becomes pinned to a deprecated but comfortable state as the deadline for hard breakage approaches.
• Semantic expectations set for the purpose of SETUPTOOLS_USE_DISTUTILS will change after over a year of stability. Perhaps a different flag could be exposed to selectively install the functionality.
• It would create divergent installs of Setuptools. So a given Setuptools version would have different behaviors depending on what environment variables were set at install time, multiplying the complexity of triage for any issue.

Else, due to 60.0.0+ changes, I need to manually add SETUPTOOLS_USE_DISTUTILS=stdlib for all OBS packaging script

Could you set that environment variable globally in the Debian environment? At least for now until Debian can adopt an approach that no longer relies on monkeypatching?

1. Keep the existing Debian-specific patching style

As distutils is going away, this approach is untenable. Setuptools is committed to adopting distutils, so if Debian wishes to own a patching approach, they would need to patch setuptools. They’re welcome to do so if that’s what they wish, but Setuptools is expecting to perform a great deal of refactoring to properly adopt distutils and remove the cruft around it, so maintaining a monkeypatch for versions of Setuptools as its evolving is going to be burdensome. That’s why this project is going out of its way to provide reliable if temporary interfaces to allow for the customizations currently provided by patches.

3. Monkeypatch style

I’m guessing by “Monkeypatch style”, you mean the approach currently proposed by this project?

Yes, this approach is somewhat complicated, but it’s less brittle than patching as done in option 1 because it specifically patches the runtime objects and not the code, so only relies on the hook points provided by distutils. Yes, it requires coordination, but since it’s meant to be temporary, I’d expect that Debian would find a way to move away from relying on these interfaces.

The primary advantage of the monkeypatch approach is that it keeps the patched behavior largely disentangled from the default implementation. That is, any Python programmer can see the entirety of the patch and disable it by deleting _distutils_system_mod or commenting out the apply_customizations() call.

I do see your point about maybe preferring option 1. I might prefer that too. Unfortunately, I don’t think Debian is interested in owning it, so I’ve supplied the system mod approach as a means of unblocking users while Debian figures out how they want to proceed. If they want to keep these behaviors long-term, they should propose the behaviors as a proper feature of the project (with designs, implementations, tests, etc).

I am not an official Debian packager for setuptools

I do regret that you’ve been caught up in this. I would have preferred that Debian maintainers would have picked this up sooner and implemented the recommended patch, tested it, and provided forward guidance to downstream packagers like yourself. I appreciate your feedback and the effort you’ve taken to understand the issue.

I welcome you or others to engage with the Debian project to devise a better solution. Right now, _distutils_system_mod is my best proposal, but it’s optional and I won’t be offended if Debian wishes to take another approach, and I offer my time and services as maintainer of Setuptools/distutils to give feedback or otherwise help guide their decisions.

---

It does feel like you’re 🤏 this close to having a working solution. Let me take a stab at extending the _distutils_system_mod to provide the deb_system and unix_system install schemes needed to honor the --install-layout. Thanks for the repro, which should make it easy for me to test.