Proposal: Add `--platform` flag to `pip install` and `pip wheel`

What’s the problem this feature will solve?

By default, the wheel platform detection pip does on Linux isn’t very advanced: it produces a platform tag like linux_x86_64, which is the same across nearly all Linux installations.

For packages with compiled bits that link against system-installed shared objects (.so files), it’s necessary to use different wheels for systems with different versions of the shared libraries. For example, a wheel built on Ubuntu 14.04 will not necessarily work on Ubuntu 16.04.

Being able to specify a custom platform name would allow building the same package version on different systems to produce two wheels with unique filenames that can be served from the same PyPI registry. For example, you could produce two wheels with the (already valid) filenames:

• numpy-1.9.2-cp36-cp36m-linux_ubuntu_14_04_x86_64.whl
• numpy-1.9.2-cp36-cp36m-linux_ubuntu_16_04_x86_64.whl

These wheels can be distributed to package consumers (e.g. internally inside a company), who get the benefits of quick wheel installation (no compile-on-install) and the possibility to work from several different platforms.

Describe the solution you’d like

We would like to add a --platform flag to the pip install and pip wheel commands, with identical behavior to the existing --platform flag on the pip download command. This change would allow building and installing wheels with a custom, user-provided platform name.

At $COMPANY, we’ve been building wheels with a custom platform name (just like the numpy example above) for several years. We upload these wheels to our internal PyPI registry, and install them using using pip-custom-platform, a wrapper around pip that adds in the --platform flag being requested in this issue.

Because we have hundreds of different internal Python projects, upgrading from one OS version to another is a long process during which we need to support development (and wheels!) on multiple OSes.

Overall, we’ve been very happy with the custom platform approach. The burden on projects and friction for developers is very low, and conceptually it fits in very nicely with the “platform” concept of a wheel (similar to how macOS wheels specify versions in the wheel’s platform tag). The downside for us right now, of course, is that we have to use a monkey-patched version of pip instead of the real thing.

pip-custom-platform has also started to be used by others in interesting use cases such as building packages for AWS Lambda to specify their own platform names.

The scope of the proposed change to pip is identical to the --platform flag for pip download: it’s just a flag that allows users to manually specify a platform to use. pip-custom-platform does have functionality to automatically generate platform names based on the Linux distribution, but we think it’s better to leave this complexity out of pip, and instead have users construct the platform name themselves and either pass in this flag manually or as an environment variable (possibly set at the system level by a system administrator, e.g. in /etc/environment).

Alternative solutions

We considered several potential alternative solutions to this problem:

• Abandon wheels entirely and compile on installation. This works but makes installation unrealistically slow (20+ minutes for some projects) and requires installing build tooling and library headers (gcc, fortran, libxxx-dev, cmake, etc.) everywhere.
• Running one PyPI registry per distribution, where each serves wheels with the linux_86_64 platform tag but which have been compiled on the corresponding platform. This could work, but is pretty unwieldy: every project would need some hackery in their build scripts to select the correct server, plus we’d have to run a ton of these registries (we currently support 3 Ubuntu versions, plus about half a dozen random platforms which are used in specialty cases but are still important to support, e.g. random Amazon Linux AMIs used for EMR). Additionally, if you were to accidentally use the wrong PyPI server and download wheels built for the wrong platform, pip would happily install them without problem and you wouldn’t notice until you get runtime errors in your code.

Additional context

Why can’t we use manylinux wheels?

manylinux wheels are a method of providing a single built wheel that works across most Linux installations. In general they do this job pretty well, but unfortunately we’re unable to use them for many packages due to the security concerns associated with them.

Specifically, for packages which need to depend on C libraries outside of the manylinux1 profile (for example, anything that depends on libssl, or almost all of the scientific Python libraries), the choices for producing a manylinux wheel are to either statically link in the dependencies, or to bundle the .so files in the wheel. In both of these cases, it is very difficult to roll out security updates across hundreds of different services, as it may involve patching and rebuilding the affected wheels, then building and deploying all of the services that consume them.

By contrast, rolling out security updates to shared libraries when services don’t bundle or statically link them is typically as easy as your distro’s equivalent of apt-get upgrade to pull in the latest patched version.

[asottile]

Requiring --target doesn’t really help our usecases described above so I’d love for an amend on that 😆

Our goal would be to do something like:

echo 'PIP_PLATFORM=linux_ubuntu_16_04_x86_64' >> /etc/environment
echo 'PIP_INDEX_URL=https://pypi.mycompany.com/simple' >> /etc/environment
echo 'PIP_ONLY_BINARY=:all:' >> /etc/environment

and then pip install x would just work

[pudquick]

I would seem to have a real world use case that’s impacted by this currently on the macOS platform.

The pyobjc project has recently published version 7.2 to PyPI. If you’re not familiar with the project, the top level module is dynamic in adjusting what its dependencies are and as such is mostly virtual / pure python while it’s the sub-modules that contain architecture specific compiled code, the base minimum required of which is actually called pyobjc-core.

If you look at how version 7.0.1 was published, you’ll see: https://pypi.org/project/pyobjc-core/7.0.1/#files

pyobjc_core-7.0.1-cp37-cp37m-macosx_10_9_x86_64.whl
pyobjc_core-7.0.1-cp38-cp38-macosx_10_9_x86_64.whl
pyobjc_core-7.0.1-cp39-cp39-macosx_10_9_x86_64.whl
pyobjc_core-7.0.1-cp39-cp39-macosx_11_0_universal2.whl
pyobjc-core-7.0.1.tar.gz

Running an install on an Intel Mac with Big Sur macOS 11 with python 3.9.2/3.9.4/3.9.5 with pip install --only-binary=:all: pyobjc-core==7.0.1, you’ll get pyobjc_core-7.0.1-cp39-cp39-macosx_11_0_universal2.whl installed - a Universal 2 copy of the module that’s safely redistributable to both Intel and Apple silicon Macs running python 3.9.2+

If we look at 7.2, though, it was published slightly differently: https://pypi.org/project/pyobjc-core/7.2/#files

pyobjc_core-7.2-cp310-cp310-macosx_10_9_universal2.whl
pyobjc_core-7.2-cp37-cp37m-macosx_10_9_x86_64.whl
pyobjc_core-7.2-cp38-cp38-macosx_10_9_x86_64.whl
pyobjc_core-7.2-cp39-cp39-macosx_10_9_universal2.whl
pyobjc_core-7.2-cp39-cp39-macosx_10_9_x86_64.whl
pyobjc-core-7.2.tar.gz

Notably the Universal 2 wheels are now tagged with macOS 10.9 instead of 11.

But the end effect is vastly different.

Running an install on an Intel Mac with Big Sur macOS 11 with python 3.9.2/3.9.4/3.9.5 with pip install --only-binary=:all: pyobjc-core==7.2, you’ll get pyobjc_core-7.2-cp39-cp39-macosx_10_9_x86_64.whl installed - an Intel-only copy of the module that’s not safely redistributable to every CPU architecture that can potentially be running macOS Big Sur.

I went through pip install verbose and didn’t find the logical reason why x86_64 was preferred over universal2 in this case, but I’m assuming it’s that “more specific is better” and that “a more specific OS version” outweighed “a more specific CPU architecture”.

For 7.0.1, there was a macOS 11 choice - but it was only universal2. For 7.2, all macOS version choices were equal (10.9+) - and both an x86_64 and universal2 option were available.

For the macOS platform going forward, at least, it highlights that there may be multiple possible valid choices - all completely compatible - but without a version of install that supports -some- level of platform filtering when it comes to choices, I can’t trivially pick the one that’s correct for my use case (in this case: putting together a self-contained environment that can run on any Mac running macOS 11 / Big Sur) and am left to a.) the somewhat mysterious logic of pip combined with b.) the python module publishing choices made by someone trying to fully support Mac.

+1 for a desire to have some sort of filtering capability when there are multiple things I could install and which one is ‘right’ is really a factor of what it is you’re trying to do.

I’ll also point out that the --target restriction also seems pretty arbitrary considering that with on extra step, I can effectively perform the desired outcome:

# make some place to put wheels
mkdir ./pyobjc-universal-7.2
# download them, filtered against what I want
python3 -m pip download --only-binary=:all: --platform=macosx_10_9_universal2 --platform=macosx_11_0_universal2 --no-cache-dir pyobjc-core==7.2 --dest ./pyobjc-universal-7.2
# only install using what I downloaded
python3 -m pip install --only-binary=:all: --no-cache-dir --no-index --find-links ./pyobjc-universal-7.2 pyobjc-core==7.2

If I can’t have the ability to pick exactly what I do want (even if it’s in the possible choices pip has at hand but doesn’t decide to use) - can I at least have the option to filter what I don’t want?