pip should support custom wheel platform tags.

As I understand it, the Wheel format has the notion of a ‘platform tag’, such that wheels built for the same version of Python with the same ABI but different platforms (say, CPython 3.4 on Windows vs. Mac OS X) will not collide. Although PEP0425 defines platform tags linux_i386 and linux_x86_64, PyPI will not accept uploads of wheels for those platforms because they are not specific enough—a linux_x86_64 wheel built on, say, Debian 8 might not work on CentOS 7, and so forth.

This is a good plan, and I support it.

However, inside the company I work for, the problem is much less complex - instead of a myriad of Linuxish platforms, we only have a handful, and we would love to be able to define custom platform tags for our internal packages, so we can be sure we’re deploying the correct wheels to the correct hosts.

To be specific, I want a workflow something like the following:

• I’m using Pip 7.0, so I have a shared wheel cache.

• My CI system notices that somebody has committed a change to my project, so it tells a member of the build-farm to run the tests.

• The build host happens to be one of our legacy CentOS 6 machines, so when it populates a virtualenv to run the tests, it runs:

  pip install --editable --platform-tag optiver_centos_6 ./source/
  

  …and so when pip consults its wheel cache, it only considers wheels whose platform tag is optiver_centos_6 or any. Furthermore, if it builds fresh wheels and caches them as part of the installation process, they will be tagged with optiver_centos_6 if they contain C extensions (or whatever the usual logic is).

• Most likely, CI system will also run the tests on CentOS 7, doing the same steps but with --platform-tag optiver_centos_7.

• Later, when my project has passed all its tests and is ready for deployment, the build host should be able to gather a complete collection of wheels compatible with the given platform:

  pip wheel --platform-tag optiver_centos_6 ./source/
  

  As before, --platform-tag should govern which wheels are pulled from the cache, as well as tagging wheels that are freshly-built.

[OptiverTimAll]

If you’d rather I bring the matter up on distutils-sig, I’d be happy to do that, but I think Pip should support custom platform tags no matter what. PEP0425 says “…built distributions should have a file naming convention that includes enough information to decide whether or not a particular archive is compatible with a particular implementation” but I believe “compatibility” is a matter of opinion.

For example, I might want to ensure that my production deployments only include wheels compiled with -D_FORTIFY_SOURCE for security hardening. Wheels built with and without such a flag might be compatible in that they both pass a test suite, but they would not be compatible if you’re looking at “likelihood to pass security penetration testing”.

Alternatively, if I have a C extension written to speed up some task, I might want to ensure that a wheel built with -march=native is only ever deployed to hosts with exactly the same micro-architecture. A wheel built for a different micro-architecture might calculate the same answer, but it might run unacceptably slowly for my purposes. Or it might even crash, if the other micro-architecture supports some extra CPU instruction.

Even if two environments should be compatible, and distutils-sig gives them the same platform-tag, I might want to distinguish them just to avoid the risk that they might somehow be incompatible in some subtle way I haven’t noticed yet.

It would be silly and unmanageable for pip to fill the platform-tag with every possible difference that could ever possibly affect anyone, especially since even the people that care probably only care about one or two of them, and it’s a different one or two for each person. However, I think it’s reasonable to say “if you have compatibility concerns that the official platform-tag system doesn’t represent, you can make up your own platform-tags as fine-grained as you want as long as you keep them to yourself and don’t upload them to PyPI”.

[dholth]

It would make sense to have something like an additional_supported_tags in the pip config file…