Setuptools and pip manage vendored namespace incompatibly

In Debian we’ve been having an issue for some time where version comparisons where one comes from pip and the other comes from setuptools are incompatible [1]. Today @dstufft helped me look into it and determined that the two tools have incompatible approaches to vendoring that are the source of this problem. Based on that insight (and very specific help, again from @dstufft ), I came up with a hack to work-around the problem for now [2].

It’s certainly not ideal, but I think it demonstrates the problem. To quote him from IRC earlier today:

what pip does is basically loop over all of our vendored modules, manually import them, and then munge sys.modules so that like, pip._vendor.packaging.version is the same object you would get back when you do import packaging.version. setuptools does it differently, they install a meta importer, that at the import level does the aliasing that effectively means that for pip, there is only one packaging module imported both at packaging and pip._vendor.packaging, the pip._vendor.packaging is just a tiny shim, whereas for setuptools packaging and pkg_resourcex.extern.packaging are two fully unique objects it’s my opinion this is an upstream bug, either in pip or setuptools (or likely both) and we’re going to need to coordinate to ensure that both of our debundling strategies can co-exist, because they currently can’t

Thanks for considering,

Scott K

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=912379 [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=912379;filename=pip.versions.patch;msg=39

[jaraco]

I’m also highly interested in removing vendoring in setuptools. It feels like to me there should be at most one tool in the python ecosystem that requires specialized bootstrapping. I’ve seen some signs that downstream packagers may be able to move to PEP 517 and pip to build setuptools from source for package manager. I’d love for this to become the standard op procedure so that setuptools could drop vendored dependencies altogether.

[jaraco]

Thanks @eli-schwartz. I’ll close it as duplicate of #1383, but I’ll flag this issue as having a lot of useful context.

FWIW, Setuptools is now discouraging the use of pkg_resources in almost every case in favor of importlib.{metadata,resources}. The __version__ could (and should) be serviced by importlib.metadata, although I’ve been discouraging use of this pattern and instead advising packages to remove this attribute in favor of having the consumer just call importlib.metadata.version() directly. In either case, use of packaging becomes explicit downstream (importlib.metadata doesn’t provide any Version objects).