Regression with Git transitive dependencies

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Linux/Mac

• Poetry version: 1.1.6

• Link of a Gist with the contents of your pyproject.toml file: https://github.com/mtraynham/poetry_4051

Issue

There seems to be a regression with transitive dependencies of a Git dependency. I haven’t exactly figured out why, but the change for the whitelist on the installer is what causes the transitive dependency to be flagged for removal rather than keeping it. The dependency is in the lock file, but when it goes to install the project, it skips it.

This particular pull causes the issue, https://github.com/python-poetry/poetry/pull/3947. Reverting the whitelist changes seems to correct the issue: https://github.com/python-poetry/poetry/pull/3947/files#diff-95c2e34175a95676a55830f8442a68da156ae6a31dfec68317a2fd746be2243fL292-L297 https://github.com/python-poetry/poetry/pull/3947/files#diff-95c2e34175a95676a55830f8442a68da156ae6a31dfec68317a2fd746be2243fL308

diff --git a/poetry/installation/installer.py b/poetry/installation/installer.py
index 8cef65d5..ebfe05db 100644
--- a/poetry/installation/installer.py
+++ b/poetry/installation/installer.py
@@ -289,6 +289,12 @@ class Installer:

         pool.add_repository(repo)

+        # We whitelist all packages to be sure
+        # that the latest ones are picked up
+        whitelist = []
+        for pkg in locked_repository.packages:
+            whitelist.append(pkg.name)
+
         solver = Solver(
             root,
             pool,
@@ -302,7 +308,7 @@ class Installer:
         solver.provider.load_deferred(False)

         with solver.use_environment(self._env):
-            ops = solver.solve(use_latest=self._whitelist)
+            ops = solver.solve(use_latest=whitelist)

         # We need to filter operations so that packages
         # not compatible with the current system,

This may actually be caused because I have 1 git dependency, which depends on another git dependency. E.g. the dependency tree looks like:

Project A -> Project B (git; poetry) -> Project C (git; setup.py) -> PyPi dep (not installed, but in the lock file)

[sdispater]

The root cause has been identified (see #4202) and this will be fixed in the next bugfix release.

Note that you will to regenerate the lock file for the fix to take effect.

[dobbymoodge]

This looks like the same issue as https://github.com/python-poetry/poetry/issues/3368

It looks like the whitelist fix referenced in the initial report is indeed the cause; it seems to have been added to prevent the solver from adding newer versions of packages than what are specified in the lockfile. It’s not clear to me, but it seems like the bug this was supposed to fix is where a package that pulls in a path or VCS dependency ignores the dependency lock file, resolves potentially newer versions of the transitive dependencies, and adds those to its own lockfile. The whitelist change prevents the solver from resolving the transitive dependencies, but instead of installing the “locked” packages, it skips them.

In 1.1.5, the solver does resolve the transitive dependencies, but it ignores the versions from the VCS/path deps lockfiles, and installs the latest version that satisfies the dependency specification from the pyproject.toml file. This is arguably incorrect, but the behavior in 1.1.6 is definitely incorrect.

Not having a firm grasp on the code, and too little time right now to debug in depth, this snippet in particular looks like it might have significance. The deferred dependencies might not need to be resolved, but they should still be loaded?

@sdispater, as the contributor of the breaking PR, what do you think? Surely the transitive package dependencies should all be installed by poetry install whether the top-level pyproject.toml specifies the dependencies as path, VCS or otherwise? This is a blocking bug for me and my team, as we rely on path dependencies for organizing our internal tooling.