pip sometimes prohibits 2 .dist-info directories in wheels

Environment

• pip version: 19.3.1
• Python version: 3.8.0
• OS: Linux

Description

Currently, pip rejects wheel files containing multiple .dist-info directories, but only if they start with the name of the package being installed.

Expected behavior

pip should prohibit more than 1 top-level .dist-info directory, without regard to the name of the directories (besides ending with .dist-info).

From PEP 427:

A wheel is a ZIP-format archive with a specially formatted file name and the .whl extension. It contains a single distribution

And from PEP 376:

One .dist-info directory per installed distribution

How to Reproduce

Save t.sh below, make it executable, and execute it.

t.sh

#!/bin/sh
cd "$(mktemp -d)"
python -m venv env
. env/bin/activate
pip install --upgrade pip wheel

echo "= Versions ====================================================================="
python -V
pip -V
wheel version

echo "= Setup ========================================================================"
echo "from setuptools import setup; setup(name='hello')" > setup.py

echo "= (1) Base case ================================================================"
echo "= (1) Build ===================================================================="
python setup.py bdist_wheel

echo "= (1) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (2) 2 .dist-info dirs (expected) ============================================="
echo "= (2) Build ===================================================================="
python setup.py bdist_wheel

echo "= (2) Add second .dist-info (shared prefix) ===================================="
mkdir hello-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl hello-example-0.0.0.dist-info

echo "= (2) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (3) 2 .dist-info dirs (NOT expected) ========================================="
echo "= (3) Build ===================================================================="
python setup.py bdist_wheel

echo "= (3) Add second .dist-info (no shared prefix) ================================="
mkdir goodbye-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl goodbye-example-0.0.0.dist-info

echo "= (3) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

Output

Output

Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: pip, wheel
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1 wheel-0.33.6
= Versions =====================================================================
Python 3.8.0
pip 19.3.1 from /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip (python 3.8)
wheel 0.33.6
= Setup ========================================================================
= (1) Base case ================================================================
= (1) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
creating hello.egg-info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
writing manifest file 'hello.egg-info/SOURCES.txt'
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (1) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0
= (2) 2 .dist-info dirs (expected) =============================================
= (2) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (2) Add second .dist-info (shared prefix) ====================================
  adding: hello-example-0.0.0.dist-info/ (stored 0%)
= (2) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 446, in run
    installed = install_given_reqs(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs
    requirement.install(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 858, in install
    self.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 487, in move_wheel_files
    wheel.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 461, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 408, in clobber
    assert not info_dir, ('Multiple .dist-info directories: ' +
AssertionError: Multiple .dist-info directories: /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-example-0.0.0.dist-info, /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-0.0.0.dist-info
= (3) 2 .dist-info dirs (NOT expected) =========================================
= (3) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (3) Add second .dist-info (no shared prefix) =================================
  adding: goodbye-example-0.0.0.dist-info/ (stored 0%)
= (3) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0

The unexpected case is (3) Install - the installation should not go through since there are multiple .dist-info directories.

[pradyunsg]

I think erroring out results in the best end-user experience here – you gave pip something invalid, so it did the wrong thing vs you gave pip something invalid and it refused to use it.

[chrahunt]

I checked all wheels associated with the most recent release of all projects on PyPI and it looks like only intel-tensorflow would be affected by this change. I couldn’t find any good contact information for that project after a few minutes of looking, since it just copies the existing tensorflow metadata. Maybe https://github.com/Intel-tensorflow/tensorflow is the repo?

In any case a possible backwards-compatible workaround:

1. upload old wheels with a new build number if needed
2. put non-primary .dist-info into {distribution}-{version}.data/purelib/