"No such file or directory" install error when egg fragment / project name differs from package name

• Pip version: 10.0.0
• Python version: 2.7
• Operating system: Ubuntu 16.10

Description:

“No such file or directory” when pip-installing with #egg=~

What I’ve run:

pip install git+https://github.com/clarkduvall/serpy.git@master#egg=TOTO

Errors I’ve got:

  Running setup.py bdist_wheel for serpy ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lE7moQ/serpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-6ovRMI --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-lE7moQ/serpy/setup.py'
  
  ----------------------------------------
  Failed building wheel for serpy
  Running setup.py clean for serpy
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lE7moQ/serpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-lE7moQ/serpy/setup.py'
  
  ----------------------------------------

My Guess

It seems that pip will git-clone the project into a tmp dir /tmp/pip-install-lE7moQ/TOTO, but when running the installation, it will look for the setup.py in /tmp/pip-install-lE7moQ/serpy/, which was not produced in pip==9.0.1

[pradyunsg]

We’d probably want to look into this as apart of #6607.

[uranusjr]

I did some debugging, here’s what I observed:

The package in question (serpy in this case) is actually installed twice. As mentioned previously, both are under the same name serpy.

The first installation correctly succeeds, but the second one generates the error because the source is deleted during the clean up phase of the first installation.

The second installation entry is added by the resolver. pip install generates a Requirement object from the argument passed to it, and adds it to a RequirementSet, which keeps Requirement objects as an OrderedDict (and other things not relevant here). This RequirementSet is passed to the resolver, which recursively discovers more dependencies, and adds more Requirement objects for those dependencies to the RequirementSet. The mismatching #egg= value, however, makes RequirementSet keep the initial Requirement under a wrong key (TOTO) during initialisation. So when the resolver processes the Requirement of serpy, it incorrectly thought it is not in the RequirementSet yet, and adds it again, this time under the correct key:

https://github.com/pypa/pip/blob/116c3b1c7ec7970481b65b886ec900c5b54f2c30/src/pip/_internal/resolve.py#L290-L297

Unfortunately, after knowing this, the fix is still not straightforward. For a remote, non-dist requirement, pip cannot know the package name before it downloads and runs setup.py (which is why #egg= is needed), but it cannot run setup.py before it knows what the package is. [*] So the #egg= specification (or the name before @ for the PEP 508 variant) really really must match the one in setup.py. So maybe the “fix” would be to hard-error when pip finds they don’t match, instead of showing a warning and failing later down the line.

[*]: Theoretically I believe it can, but that would probably require a huge restructure to implement.