Package source priority (HPC cluster use case)

_Originally posted by @mboisson in https://github.com/pypa/pip/issues/8606#issuecomment-877431454_

Here are what we can and can not do, in our environment (HPC clusters).

1. We can’t run a server. This is for HPC clusters which are not under our control. The only thing we distribute is a filesystem. We can’t long-standing processes and we don’t control the network.
2. We can’t run binary (manylinux or other) wheels from PyPI for two primary reasons reasons: missing libraries or libraries installed in non-standard locations, non-optimized CPU instructions
3. We can’t host all of PyPI, that’s just too much, and pure-python wheels work just fine.
4. We can’t change our requirement files/install differently, because we aren’t the ones doing the installation. We support end users (researchers), we don’t do their works, they install whatever package they need.

What we can do, and have been doing:

1. Provide users a directory with wheels that we compile from source
2. Define a PIP_CONFIG_FILE in the user’s environments to make sure their pip install ... commands use some settings. We have been using this to point their pip to our wheel directory, which used to be preferred over pypi. We are now also using that to put a constraint on pip < 21, since that’s now broken for us.

[uranusjr]

Wheels on PyPI can’t have build tags, IIRC.

I believe build tags are possible on PyPI, although practically nobody uses them.

[uranusjr]

By two packages with the same label, do you mean e.g. there are two pages for package foo, or there are two entries for file foo-1.0-none-any.whl? Because the former is an absolutely valid strategy to support architectures that PyPI doesn’t and potentially can’t host, and is used by many projects like piwheels and pytorch. Showing a warning would cause great disruption to those projects (because unassuming users would bug them thinking they are to be faulted), who have been meticulously following community guidelines and doing things “right”. The latter is likely harmless, but probably also not very useful?