pip breaks when pip installs and conda uninstalls the same package

Environment

• pip version: 20.2.4
• conda version: 4.9.2
• Python version: 3.8.6
• OS: linux

Description

This is a bug in pip, but it gets triggered by conda.

When installing a package explicitly it creates an empty file REQUESTED, which conda can’t delete and everything blows up.

Expected behavior

Since conda removes explicitly the files in its manifest unlike pip which removes the whole dir, adding new files that are not in manifest breaks things.

How to Reproduce

# cleanup
conda uninstall -y pynvml
pip uninstall -y pynvml

# must install/uninstall things in this exact order
conda install -y -c conda-forge pynvml
pip uninstall -y pynvml
pip install pynvml
conda uninstall -y pynvml

# check: not good:
ls -lt /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/pynvml-8.0.4.dist-info/
total 0
-rw-rw-r-- 1 stas stas 0 Nov 12 22:03 REQUESTED

This file wasn’t in conda-forge build, but was added by pip - conda doesn’t know to remove it since it removes files explicitly only the ones it installed, but pip removes the dir.

So now the site-packages is broken and we can’t use pip anymore:

pip install pynvml
Requirement already satisfied: pynvml in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages (8.0.4)
WARNING: No metadata found in /mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/pynvml-8.0.4.dist-info/METADATA'

“Requirement already satisfied” is bogus since there is just one REQUESTED file that wasn’t cleaned up in the dist dir.

this fails too:

pip uninstall -y pynvml
Found existing installation: pynvml 8.0.4
ERROR: Exception:
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/commands/uninstall.py", line 89, in run
    uninstall_pathset = req.uninstall(
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 685, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_uninstall.py", line 535, in from_dist
    for path in uninstallation_paths(dist):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_uninstall.py", line 67, in unique
    for item in fn(*args, **kw):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_internal/req/req_uninstall.py", line 85, in uninstallation_paths
    r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1432, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages/pynvml-8.0.4.dist-info/RECORD'

The only fix at the moment is to manually remove the dir by hand.

I have a package whose build process validates that the conda/pip installs worked correctly and so it happens to hit a similar sequence. But this can easily happen in real life, except install / uninstall commands can be called days apart…

[pfmoore]

No problem! Thank you for taking the time to dig into this - the “METADATA issue” #8676 has been a long-running problem for us, and it’s great to understand what was going on here and finally identify how it needs to be fixed!

[uranusjr]

Ah, so this is the cause to #8676, eh? We’ve had trouble reproducing this and you’re the first to identify a likely cause. Thanks for filling in the missing piece!

This looks like a Conda bug. pip needs RECORD to know what files to uninstall. If Conda removes the file, it should also remove the .dist-info directory, which pip relies on to discover installed packages. It should not uninstall the package’s metadata half-way.

pip can’t really do anything here since uninstallation cannot be reasonably done without RECORD. IMO this is a Conda bug and should be fixed by Conda.