Running poetry in parallel fails on file access

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04 on WSL
• Poetry version: 1.0.0a4
• Link of a Gist with the contents of your pyproject.toml file: n/a

Issue

While setting up tox based testing with Poetry, I kept getting odd errors about a missing pyproject.toml file. A minimal reproduction is

python -m virtualenv .test/py36 --python python3.6
python -m virtualenv .test/py37 --python python3.7
VIRTUAL_ENV=.test/py36 poetry install -vvv --no-dev || echo "failed 36" &
VIRTUAL_ENV=.test/py37 poetry install -vvv --no-dev || echo "failed 37" &
wait

This will get one of the processes erroring out on the last step with

[FileNotFoundError]
[Errno 2] No such file or directory: '/home/pbecotte/PycharmProjects/rye/pyproject.toml'

Traceback (most recent call last):
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 112, in handle
    status_code = self._do_handle(args, io)
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py", line 160, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/home/pbecotte/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/home/pbecotte/.poetry/lib/poetry/console/commands/install.py", line 96, in handle
    builder.build()
  File "/home/pbecotte/.poetry/lib/poetry/masonry/builders/editable.py", line 17, in build
    return self._setup_build()
  File "/home/pbecotte/.poetry/lib/poetry/masonry/builders/editable.py", line 38, in _setup_build
    str(self._poetry.file), str(self._poetry.file.with_suffix('.tmp'))
  File "/usr/lib/python3.7/shutil.py", line 577, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.7/shutil.py", line 263, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

From following the stack trace, I can see that Poetry is temporarily renaming the pyproject.toml file to trick pip into doing an editable install, which is causing the second process to fail.

I’d be happy to add a PR to fix this, but am kind of lost on the right approach. Should there be a lock/wait before the file rename (can you use a file lock on a file that gets renamed?)? Is there a way to directly invoke pip to do the right thing without moving any files? Should we use the previously suggested approach of invoking setuptools directlly? Should I just suck it up and put retry logic into my Makefile?

[franknarf8]

It seems you can use the --no-root flag of the poetry install command to avoid this concurrency issue. tox will install the root package anyway if you do not specify the install skip_install setting.

Something of the sort :

[testenv]
whitelist_externals = poetry
commands =
    poetry install -n -v --no-root
    poetry run pytest [...]

[johnfraney]

I came up against this today using the most recent version of Poetry, v1.0.0b9.

My workaround was to avoid using poetry install to install dependencies, and instead to use poetry export to generate a requirements file and pip install to install dependencies from that requirements file. This allows Tox to work in parallel because no files are renamed when installing dependencies.

Here’s my tox.ini for reference:

[tox]
skipsdist = True
envlist = py36, py37, py38, pypy3

[testenv]
whitelist_externals = poetry
skip_install = true
commands_pre =
    poetry export -f requirements.txt -o .tox/requirements.txt
commands =
    pip install -r .tox/requirements.txt
    pytest

I hope this helps!