Poetry install with version 1.1.6 fails with file not found errors

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian GNU/Linux 10 (buster), as well as macOS Catalina 10.15.7 (19H2)
• Poetry version: 1.1.6 (although seen on 1.1.5 too)
• Link of a Gist with the contents of your pyproject.toml file: One such project experiencing this issue is https://gist.github.com/lucinvitae/d9faa08f9bf9f151aa2032212443ecda but it appears to affect all of our projects using 1.1.6 regardless of pyproject.toml format.

Issue

Poetry install failures with FileNotFoundError exceptions occur for version 1.1.6 and are occasionally blocking the ability to build/deploy our internal apps. This occurs for multiple of our company’s projects that install dependencies with poetry. The issue has been reproduced with poetry 1.1.6 and 1.1.5. It occurs locally outside of docker, locally in docker, and in CI using docker. It is sporadic, not affecting every build every time, and thus difficult to reproduce (requires running builds over and over in parallel). Due to the intermittent nature, this typically hits us in CI.

Workaround: We are able to successfully workaround this poetry install flakiness by using Poetry version 1.0.10. Right now we have pinned poetry at that version and are unable to upgrade without seeing significant CI / CD instability. So, we are blocked from using poetry’s latest features at scale.

Example failure error:

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/site-packages/~11-0.9.0.dist-info'
...
#13 ERROR: executor failed running [/bin/sh -c poetry install --no-root]: runc did not terminate sucessfully

Several similar FileNotFoundError tickets exist but are for the poetry run command instead of the poetry install command` and do not seem like the same issue:

• https://github.com/python-poetry/poetry/issues/1567
• https://github.com/python-poetry/poetry/issues/3592

We were able to collect several verbose (-vvv) logs for example intermittent failures, which do not occur when retriggering the build. Logs:

poetry-filenotfounderror-1.log poetry-filenotfounderror-2.log

The exceptions, such as the one in the code block above, typically feature some filename that appears to be truncated (e.g. No such file or directory: '/usr/local/lib/python3.8/site-packages/~11-0.9.0.dist-info. Another example seen recently when running a local macOS Catalina environment:

• Updating typed-ast (1.4.1 -> 1.4.3): Failed

  EnvCommandError

  Command ['/Users/myuser/w/ai/ai-vision-utils/venv/bin/pip', 'install', '--no-deps', '-U', 'file:///Users/myuser/Library/Caches/pypoetry/artifacts/2a/97/67/6a4e444fede37f31a09025fea88c54d4a40d739c4640e72d4c8d355661/typed_ast-1.4.3-cp38-cp38-macosx_10_9_x86_64.whl'] errored with the following return code 1, and output:
  Looking in indexes: https://my.internal.pypi.repo/repository/invitae-pypi/simple/, https://pypi.python.org/simple/, http://build.locusdev.net:10001/simple/
  Processing /Users/myuser/Library/Caches/pypoetry/artifacts/2a/97/67/6a4e444fede37f31a09025fea88c54d4a40d739c4640e72d4c8d355661/typed_ast-1.4.3-cp38-cp38-macosx_10_9_x86_64.whl
  Installing collected packages: typed-ast
    Attempting uninstall: typed-ast
      Found existing installation: typed-ast 1.4.1
      Uninstalling typed-ast-1.4.1:
        Successfully uninstalled typed-ast-1.4.1
  ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/Users/myuser/w/ai/ai-vision-utils/venv/lib/python3.8/site-packages/~lick-7.1.2.dist-info'

Note again, the trimmed filename (~lick-7.1.2.dist-info). Rerunning the command fixes the issue, similar to re-triggering the build in CI when we use poetry 1.1.6, which we don’t anymore due to this flaky install issue.

Any help would be really appreciated, because we use 1.1.6 locally for some test projects, and really like it otherwise! 🙏

[lucinvitae]

Posting an update to help others with this issue, as well as maintainers that may attempt to debug this issue.

A member of our team has discovered a better workaround for this issue than downgrading from 1.1.6 to 1.0.10: https://python-poetry.org/blog/announcing-poetry-1-1-0.html#brand-new-installer

Disabling the new poetry installer by running the following command appears to solve the CI/install flakiness issue for us:

poetry config experimental.new-installer false

Breadcrumb trail, see this comment: https://github.com/python-poetry/poetry/issues/3219#issuecomment-714345861

[neersighted]

Closing for now due to lack of reproduction and old versions being reported.