Wheel installation fails when local path dependency introduced

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MAC OS BigSur

• Poetry version: Poetry version 1.1.12

• Reproducible code: https://github.com/stonelazy/poetry-path-issue

• Issue

  • .whl file generated using poetry build throws error when trying to install.

  • Steps to reproduce this issue

    git clone https://github.com/stonelazy/poetry-path-issue cd poetry-path-issue poetry build pip install dist/appname-1-py3-none-any.whl -vvv

  • Expected result - whl gets installed without any error.

  • Issue faced - Exception trace thrown when invoking pip install of the .whl file

  • Issue Description - This issue occurs only when a another local path dependency is kept. Suppose the line L13 is removed, then there is no issue. - Including of this local path is correct according to https://python-poetry.org/docs/dependency-specification/#path-dependencies

ERROR: Exception:
Traceback (most recent call last):
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
    return self.__dep_map
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3101, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/packaging/requirements.py", line 117, in __init__
    raise InvalidRequirement(f"Invalid URL: {req.url}")
pip._vendor.packaging.requirements.InvalidRequirement: Invalid URL: appname-pkg3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
    status = self.run(options, args)
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 203, in wrapper
    return func(self, options, args)
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 315, in run
    requirement_set = resolver.resolve(
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 366, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 212, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 202, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 197, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 197, in <listcomp>
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 250, in iter_dependencies
    requires = self.dist.requires() if with_requires else ()
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2736, in requires
    dm = self._dep_map
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3023, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3033, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/Users/sudharsan-2598/opt/anaconda3/envs/npsp6/lib/python3.9/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pip._vendor.pkg_resources.RequirementParseError: Invalid URL: appname-pkg3

• Tree of the repo

├── README.md
├── appname
│   └── pkg1
│       ├── __init__.py
│       └── sample.py
├── appname-pkg3
│   ├── appname
│   │   └── pkg3
│   │       ├── __init__.py
│   │       └── sample.py
│   └── pyproject.toml
├── dist
│   ├── appname-1-py3-none-any.whl
│   └── appname-1.tar.gz
├── error.txt
├── poetry.lock
└── pyproject.toml

• Expected Result:

  • Installation gets completed and that i would be able to do following the python import.

  from appname.pkg3 import sample
  from appname.pkg1 import sample
  

• Why should this be a poetry issue and not something wrong with the code ?

  • If i do poetry lock and then poetry install then i don’t face any issue.

[KholdStare]

Currently setting up a monorepo with python services/libraries, and need to work around these current limitations.

At the end of the day, poetry supports local paths for dependencies so I think there needs to be a sane outcome when running poetry build in this case. IMHO, it should most likely package all the local dependencies up into the built .whl file - that’s the simplest solution. If something is local, and we’re building a module to be installed somewhere, it makes sense to bundle any non-external dependencies with the build, because there is no other way to get them.

I mentioned the same in https://github.com/python-poetry/poetry/issues/1168

[TBBle]

I was actually thinking that poetry build would be doing the first two dot points, because when it looks at the pathed dependency, it can see the version of the pointed-to package generate that == dependency as part of the resulting setup.py pretty trivially.

I agree that it’s not a workflow for every occasion: It works well for “I run poetry build on every package in parallel in my CI system, upload them to a directory, and then COPY that directory into a DockerFile where I call pip install for the particular top-level packages being installed in this case”, but vendoring as you describe is a better workflow for poetry build -o X.whl && pip install X.whl for a particular top-level package X.

The flow I want to use also works for “I build each package and push it to our PyPI repo in parallel”, particularly if there are consumers of the libraries (not the root packages) who’re not in the same monorepo, and hence need a normally exported package anyway. This flow means that both the monorepo output and the external users are using the same wheels.

That said, if vendoring into a single wheel is implemented and reliable, I’d probably use it in preference to the current approach, which is exposing all the source into the containers and poetry install there. It’d be a significant improvement, even if it’s not my perfection vision.

I’ll also note that one of my considered approaches was to only use dev-dependencies in pathed dependencies, and then use normal versioned package dependencies in the non-dev dependencies. In this case, I’d want poetry build to ignore the dev-dependencies (which is a different problem…) and poetry install to fail if the two dependencies for the package conflict (i.e. what’s at the path has a current version that doesn’t meet the version requirement).

In this flow, it’s required (and expected) that people are maintaining the versioned-dependency (dot point 2), and intentionally so, as that version is an ABI dependency, and should be being bumped when ABIs are changed/added/removed, so you can’t successfully install mismatching packages.

However, Poetry has some other limitations that currently prevent this, i.e. the resolver merges dependencies and dev-dependencies too early to make this work, if I recall correctly. (There’s a ticket elsewhere here tracking this use-case, so it’s not specifically-relevant here, I just wanted to share some context on my approach here.)