Handle multiple concurrent pip processes on the same environment

I saw that pip 6 added file-based locking to the http cache to avoid multi-process contention. Something should be done to handle multiple packages being installed into the same virtualenv at the same time. For example if I had a concurrent makefile that installed independent foo_webapp1 and foo_webapp2 packages with a common dependency on foo_core, what would happen right now? What should happen? I guess the easiest is just to abort the second process.

[pradyunsg]

Some of our tests for PEP 561 require pip to install packages, so we’re recently run into some instability in our test suite. Note that each copy of pip is installed in its own venv.

This should still work. Could you file an issue with the failure logs and a reproducer for the behaviour you’re seeing?

Multiple invocations of pip in separate environments should work already. That’s literally what pip’s own test suite does.

[ethanhs]

In mypy we run our test suite in parallel. Some of our tests for PEP 561 require pip to install packages, so we’re recently run into some instability in our test suite. Note that each copy of pip is installed in its own venv. Sadly it seems that even with --cache-dir or --no-cache there is some race happening that seems to cause the tests to fail intermittently.

For now we will likely put a lock around package install, but it would be nice if @pradyunsg’s suggestion of throwing an error if pip is already running would be implemented, it would certainly have made it easier to discover what was going wrong with our tests 😃

What would the next steps be to get to concurrent pip? I imagine a list of all the places pip relies on global state would be a good start.