python tag issue with pip wheel / pip install / caching

A tricky issue I discovered while re-reading WheelBuilder.build.

I got to wonder why the python tag of the built wheel would depend on whether we were doing pip install (should_unpack=True) or pip wheel (should_unpack=False).

With pip 19.3.1, do this (the chosen distribution name is a pure python sdist that has no wheel PyPI):

• pip install odoo-autodiscover (the wheel is built and cached)
• pip wheel odoo-autodiscover (the wheel is obtained from cache: odoo_autodiscover-2.0.0-cp37-none-any.whl)

Notice the cp37 tag.

Now, empty the cache and do:

• pip wheel odoo-autodiscover (the wheel is built and you get: odoo_autodiscover-2.0.0-py3-none-any.whl)

Notice the py3 tag.

So depending on whether the wheel was installed and cached before, the result of pip wheel is different.

This behaviour was apparently introduced in 0e240d7ddeb28c73d2906657eab7c7a215747f4d.

Now with systematic caching introduced in #7285, the behavior is subtly different: a py3 or cp37 wheel gets cached depending on whether pip install or pip wheel is executed first. Whereas before #7285, only cp37 could be cached, never py3.

I’m not too sure what to do with that yet, as I don’t really understand the motivation of 0e240d7ddeb28c73d2906657eab7c7a215747f4d.

[chrahunt]

I agree that projects should be using the existing facilities (e.g. python_version and implementation_name markers) to specify conditional dependencies.

I have a few concerns with removing the interpreter-specific wheel cache outright:

1. the failures will happen at runtime and will be hard to connect to the wheel-building/caching because of conditional dependencies in setup.py and conditional imports in code. I can’t think of a way other than “check the setup.py at the first sign of any issue” to help identify that a problem is caused by a non-conforming package.
2. I can’t see any good place to warn users about this. Building any legacy project into a wheel (or any project that uses a backend which produces metadata nondeterministically but doesn’t update wheel tags) may produce a wheel that is not actually portable, but warning for all of them seems like it would be overkill.

[xavfernandez]

If 1. is considered too risky, we could combine it with having a separate wheel cache root directory per python implementation.

seems the cleanest to me but would require some thinking on the cache layout transition.

Edited: and I’ve just seen that #7319 has been created 😫 Don’t mind me 😖