Poetry install fails for nested local dependencies and develop = false

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Standard Python:3.8 docker image
• 1.1.4

Issue

I am attempting to install a poetry based application with a monorepo-like hierarchy. For simplicity sake the structure looks something like:

/lib
    /pkg1
        pyproject.toml
    /pkg2
        pyproject.toml
/apps
    /app1
        pyproject.toml

The apps/app1/pyproject.toml is completely empty other than a reference to one of the libraries: pkg1 = { path = "../../lib/pkg1", develop = false }

pkg1 has then a similar local dependency on pkg2: pkg2 = { path = "../pkg2", develop = false }

An example repo of this is here: https://github.com/TrevorPace/poetry-localdep-issue

When attempting to run poetry install in apps/app1 to create the initial poetry.lock file I get the following issues:

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing pkg2 (0.1.0 /home/trevor/git/trevor.pace/poetry-localdep-issue/libs/pkg2)
  • Installing pkg1 (0.1.0 /home/trevor/git/trevor.pace/poetry-localdep-issue/libs/pkg1): Failed

  EnvCommandError

  Command ['/home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/bin/pip', 'install', '--no-deps', '-U', '/home/trevor/git/trevor.pace/poetry-localdep-issue/libs/pkg1'] errored with the following return code 1, and output:
  Processing /home/trevor/git/trevor.pace/poetry-localdep-issue/libs/pkg1
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/bin/python /home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp8olvb_pd
           cwd: /tmp/pip-req-build-845k8blz
      Complete output (16 lines):
      Traceback (most recent call last):
        File "/home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
          main()
        File "/home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-q6c1rb7q/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 34, in prepare_metadata_for_build_wheel
          poetry = Factory().create_poetry(Path(".").resolve())
        File "/tmp/pip-build-env-q6c1rb7q/overlay/lib/python3.8/site-packages/poetry/core/factory.py", line 91, in create_poetry
          self.create_dependency(name, constraint, root_dir=package.root_dir)
        File "/tmp/pip-build-env-q6c1rb7q/overlay/lib/python3.8/site-packages/poetry/core/factory.py", line 242, in create_dependency
          dependency = DirectoryDependency(
        File "/tmp/pip-build-env-q6c1rb7q/overlay/lib/python3.8/site-packages/poetry/core/packages/directory_dependency.py", line 36, in __init__
          raise ValueError("Directory {} does not exist".format(self._path))
      ValueError: Directory ../pkg2 does not exist
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/bin/python /home/trevor/.cache/pypoetry/virtualenvs/app1-Jm016c7i-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp8olvb_pd Check the logs for full command output.


  at ~/.local/lib/python3.8/site-packages/poetry/utils/env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│
      1076│         return decode(output)
      1077│
      1078│     def execute(self, bin, *args, **kwargs):

So, it appears that we are actually able to correctly traverse the package tree when determining what to build, but when pkg1 actually is being built it’s generating something with a local reference in it still. My initial thought is that it is somehow related to https://github.com/python-poetry/poetry/issues/3148.

[dobbymoodge]

Resurrecting this discussion as this bug is still current, and I suspect it may be related to #4051, and @sinoroc expresses some preference here which I think needs to be repudiated:

I’m not a maintainer, but I will try to look at this again, see if I can pin point more accurately what is going wrong here.

Digression:

I believe the potentially surprising and/or unreliable behaviour becomes quite clear with more knowledge of Python’s packaging story in general (not just poetry). In my mind it is clear that things like path dependencies are not first class citizens at all. And from my point of view I also do not know why they should be. If I could I would probably get rid of them entirely.

… etc.

Whether you like path dependencies or not is totally irrelevant to the bug in question. Poetry is supposed to be able to resolve dependency hierarchies recursively, and it should do so correctly for each kind of dependency it is designed to support.

In this instance, pkg2 is being installed correctly, but when pkg1 dependencies are being resolved at install time, poetry fails to synthesize a package name from the path dependency in pkg1’s pyproject.toml. The develop = false flag should instruct poetry to express the package dependency as a name instead of a path when feeding pip for pkg1 - that way pip would find the matching package amongst the installed packages, mark the dependency as satisfied, and continue as normal. This is not an intractable problem.

Poetry should be able to handle these kinds of dependencies - it’s central to the design goals of the tool. Breaking when develop = false isn’t acceptable; why take a boolean if only one value will work? With the path and the pyproject.toml, Poetry has enough information to correctly resolve the dependency and provide pip with the information it needs to successfully install pkg1.

You don’t understand path dependencies or don’t think they’re “nice”, or whatever. That doesn’t mean that this isn’t a bug. Spending paragraphs of italicized text explaining that you don’t understand something is distracting. You explain that you don’t even understand the “monorepo” use case - if you don’t understand the use case, and don’t understand the failure, then don’t make the issue difficult to approach by building walls of text on top of each other.

This bug affects Poetry 1.1.6.

[TrevorPace]

@malte-klemm-by

I’m not a maintainer, but I will try to look at this again, see if I can pin point more accurately what is going wrong here.

Digression:

I believe the potentially surprising and/or unreliable behaviour becomes quite clear with more knowledge of Python’s packaging story in general (not just poetry). In my mind it is clear that things like path dependencies are not first class citizens at all. And from my point of view I also do not know why they should be. If I could I would probably get rid of them entirely.

I guess I see how path dependencies could be somewhat useful for things like “mono-repo”, although I do not have much knowledge about those (seen from afar I do not see the advantages of mono-repos). Anyway, I do not understand why mono-repos want to bypass the distribution phase. Mono-repo or not, my impression is that it should still be possible to build the individual projects and upload them to an index (PyPI or private repository), and thus get rid of path dependencies.

Also I must admit I do not really understand why people bother with building individual projects in a mono-repo. This all seems quite counter-productive, a weird mix of things, that result in doing many things that are against best practices. Why do you need dependency resolution in a mono-repo? Aren’t all dependencies supposed to be in the repo, even the external ones? All these things do not make sense to me. Why go for a radically different approach (mono-repo) and then complain that tools do not work with this approach? I always assumed that mono-repos used radically different tools. I am curious about these things.

There is a bit of work in progress that could potentially help with these kinds of project structure, maybe it is of interest to you:

• #2270
• #936

To address your comments on the mono-repo: There are significant advantages for a mono-repo when it comes to closed-source development of microservice-based projects. When it comes to development you are able to literally change files in the dependent libraries on the fly without having to first build/push to a private repo. By having all of your components in one repo it promotes quicker code sharing as well removes the need for any sort of “manual” versioning of the various components. As a result, you simply deal with one repo and merging the branches from it. A refactor, or split of services would not require removing or adding other repos, but instead be completely isolated to the branch. When it comes to deployment you can basically use git describe tags to get a unique version identifier for all the artifacts in that specific commit. Meaning that you guarantee all of the individual services are functionally aligned, there is no thinking about “What version does service X have to be with service Y?”. Working with private repos is pretty painful when it comes to making CI/CD pieplines as well as docker images.

When it comes to python/poetry, ultimately what we are looking for is a way to do development locally (IE install into a virtual environment for a given service) in a way that local paths are used, but for an install (into something like a Docker container) we want to be creating actual packages for each component in the dependency tree and installing them into site-packages. NPM for example doesn’t allow local development via linked resources, but instead when you install a local dependency it copies it (and all other dependencies) into your source tree. That makes it easy for deployment, but for development you need to rerun the install step after making any changes to the local dependency services or libraries.

My general opinion is that the whole use of the “develop” flag within the actual pyproject.toml file is counter-productive. If anything we should have a flag passed when running “poetry install” like “–local-dev” or similar. This way without the flag all packages would be copied into the virtual environment correctly with all local path dependencies resolved to just package names (similar to npm install or what we need for docker file packaging), but with the flag the package links (egg-link I think) would be installed into the virtual environment instead.

Ultimately, what we are trying to do is stack a hierarchy of dependencies and either let them be there for development, or flatten them into standalone packages based on what we actually need.