Parallell pex creation fails due to possible race conditions in package installation

We’re using pex with makefiles and parallell jobs in CI and are noticing spurious errors like the one below

pex --find-links out/wheels -o out/pexes/pex1.pex ./pex1 -e pex1:main
pex --find-links out/wheels -o out/pexes/pex2.pex -r pex2/requirements.txt --python-script pex2/entrypoint.py
Could not find the installation RECORD for zipp 3.10.0 under /root/.pex/installed_wheels/4fcb6f278987a6605757302a6e40e896257570d11c51628968ccb2a47e80c6c1/zipp-3.10.0-py3-none-any.whl.workdir

The other pex built fine and if I retry it will work. Problem is probably made worse by having all wheels cached so no time is spent downloading them which makes it more likely to run into races when installing.

This would probably need some locking around the installed_wheels folder or more fine grained locking/atomik operations for the real installation folder.

It’s not urgent for us as I can avoid making those in parallell or ensure separate pexdirs but it was somewhat of a nasty issue to debug and would perhaps save someone the time and frustration in the future.

[jsirois]

I repro - which is great. Thank you for setting that gist up. I should be able to get to the bottom of this pretty quickly.

[jsirois]

Actually - one question presents itself. Are your errors confined to missing files in paths that are prefixed by /root/.pex/installed_wheels/? If so, that might be a special case I can look at. I think that is the oldest use of atomic_directory and it may not use the locking at all, just an atomic rename. Your examples indicate this happens when Pex is calling pip install (Pex dogfoods itself and runs Pip as a venv PEX in there).