pip 21.3+ in-tree builds with setuptools, do not clean local build directories

Description

Given a requirements.txt including a local package path, installing them will litter the local path with in-tree build folders.

This is slightly bloating docker builds and causing some issues due to build artifacts permissions that force us to clean every local build folder with

WORKDIR /path/to/package-1
RUN python setup.py clean --all
WORKDIR /path/to/package-2
RUN python setup.py clean --all
# etc

but note this won’t slim image sizes either.

Expected behavior

Previous pip versions built local packages in folders like /tmp/pip-req-build-XXXXXXX/ . Regardless of being in-tree or out-of-tree, the point is those folders vanished at the end of a pip install instruction unless the --no-clean flag was used.

pip version

21.3

Python version

3.9.7

OS

GNU/Linux + Docker 20.10.8

How to Reproduce

1. git clone https://github.com/pypa/sampleproject.git /tmp/sampleproject
2. pip install --use-feature=in-tree-build /tmp/sampleproject
3. find /tmp/sampleproject/build

Output

/tmp/sampleproject/build
/tmp/sampleproject/build/bdist.linux-x86_64
/tmp/sampleproject/build/lib
/tmp/sampleproject/build/lib/sample
/tmp/sampleproject/build/lib/sample/__init__.py
/tmp/sampleproject/build/lib/sample/simple.py
/tmp/sampleproject/build/lib/sample/package_data.dat

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[sigma67]

This issue caused a lot of confusion for me, as repeated in-tree installs with pip and pyproject.toml would install the prior version in the build directory instead of the local files with my changes. I wasn’t aware that such a directory was being left behind.

It would be highly appreciated if setuptools did not leave behind a build directory.

[pfmoore]

We’ve been round this loop a few times now. Incremental builds are another topic that comes up in this context. When PEP 517 was being designed, there was a lot of discussion over how to control what the backend did, and ultimately the strong consensus was to not try to force backends to behave in a particular way, but to “trust the backend” to do the build. Being able to do in-place builds in pip was also a motivating factor in these discussions, as we only really did out-of-place builds to protect against backend issues (stale build artefacts affecting the build, for example).

In practice, backends (in particular, setuptools) haven’t changed much since those original discussions, and so the various issues that were discussed remain. I suspect pip’s out-of-place build mechanism removed a lot of the incentives for backends to consider the problem. So we’re now in a situation where the implications of the “trust the backend” principle are becoming more apparent, but there will be a transition period before mechanisms appear in backends and/or user workflows to address use cases like this.