Install on Heroku fails with pip error

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name:

• Poetry version: 1.1.7

• pyproject.toml

Issue

We use Poetry on Heroku to install our packages. For this, we list poetry==1.1.7 as sole dependency in our requirements.txt, and have a post-compile hook that looks like this:

python -m poetry config virtualenvs.create false
python -m poetry install --no-dev 2>&1

Additionally, we invalidate the cache in the pre-compile hook to ensure we always start with a clean environment and don’t have any old packages lying around:

echo "taint cache" > "$CACHE_DIR/.heroku/requirements.txt"

All in all this seems to work fine, however, we spuriously encounter errors like these:

-----> Running post-compile hook

-----> Installing dependencies with Poetry...

      Skipping virtualenv creation, as specified in config file.

      Installing dependencies from lock file

      

      Package operations: 91 installs, 7 updates, 4 removals

      

        • Removing packaging (20.9)

        • Removing pexpect (4.8.0)

        • Removing ptyprocess (0.7.0)

        • Removing pyparsing (2.4.7)

        • Updating six (1.16.0 -> 1.15.0)

        • Updating cffi (1.14.6 -> 1.14.5)

        • Installing jmespath (0.10.0)

        • Installing python-dateutil (2.8.1)

        • Updating urllib3 (1.26.6 -> 1.26.5)

        • Installing botocore (1.20.34)

        • Updating certifi (2021.5.30 -> 2020.12.5)

        • Installing chardet (3.0.4)

        • Updating cryptography (3.4.7 -> 3.4.6)

        • Installing defusedxml (0.7.1)

        • Installing et-xmlfile (1.0.1)

        • Updating idna (3.2 -> 2.10)

        • Installing multidict (5.1.0)

      

        EnvCommandError

      

        Command ['/app/.heroku/python/bin/python', '-m', 'pip', 'install', '--no-deps', '/app/.cache/pypoetry/artifacts/36/92/90/7beba42c3a6efbfdc30dc864c33286fa23817b3942b21ccaf319f18ed3/multidict-5.1.0-cp38-cp38-manylinux2014_x86_64.whl'] errored with the following return code 1, and output: 

        Processing /app/.cache/pypoetry/artifacts/36/92/90/7beba42c3a6efbfdc30dc864c33286fa23817b3942b21ccaf319f18ed3/multidict-5.1.0-cp38-cp38-manylinux2014_x86_64.whl

        Installing collected packages: multidict

        ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/build_8c19dde7/.heroku/python/lib/python3.8/site-packages/~dna-3.2.dist-info'

        

        WARNING: You are using pip version 20.2.4; however, version 21.1.3 is available.

        You should consider upgrading via the '/app/.heroku/python/bin/python -m pip install --upgrade pip' command.

This error is not specific to a single package, we saw it happen for different packages.

The error always is the No such file or directory for the .dist-info.

The file/directory that is not found always has the first character substituted by ~ which might be a name constructed by pip’s AdjacentTempDirectory.

As the error is only occurring spuriously and as I have no interactive access to the build environment, I am not really sure how I should best debug this further.

[cfra]

@finswimmer this is a build on Heroku, not an interactive environment.

I cannot modify the environment and rerun from a specific step, I can just rerun the complete build.

Also, I cannot delete the ~dna-3.2.dist-info directory, as the directory is not there, as can be seen from the “No such file or directory error” in the log.

As for where the ~ is coming from:

Pip uses a class AdjacentTempDirectory to shuffle things around when it installs packages.

That class creates a temporary directory adjacent to another path.

For this, it replaces the first character of the path with ~.

Specifically, looking at my error log above, AdjacentTempDirectory will create a directory ~dna-3.2.dist-info when it is asked to create a directory adjacent to idna-3.2.dist-info.

Also, note that immediately prior to the error that happens when attempting to install multidict, we are updating idna.

So my assumption would be that the update and the install are being run in parallel which is apparently unsafe.

I think the pip updating idna creates a temporary directory ~dna-3.2.dist-info that is collected by the pip installing multidict, however the pip updating idna removes/renames that directory so that it just “vanishes” from the perspective of the pip installing multidict, leading to the “No such file or directory error” that can be seen above.

[uSpike]

I mentioned it in #4568 but I am seeing this issue as well. @cfra analysis looks sound. It seems that executing multiple pip operations in parallel where packages have shared dependencies is not safe for parallel execution.