Performance is dramatically worse for `-r requirements` than without it.

Bug description

It is very slow.

Reproduction steps

pip-audit vs pip-audit -r requirements.txt and the -r version is unbearably slow (I’ve never seen it finish.) So I’ve repro’d the hang in alpine and OFF VPN, so my .pypirc and VPN isn not the cause.

Clone this. https://github.com/matthewdeanmartin/pip_audit_in_docker/

FROM python:3.9-alpine
RUN pip install pip-audit
RUN mkdir -p app
# Thrashing, not sure why it fails
RUN chown root:root /app
RUN chown root:root /tmp
# pip-audit will install ccfi on 1st run, needs C++ etc.
RUN apk add --no-cache libffi-dev build-base
WORKDIR /app
ENTRYPOINT ["pip-audit"]

Build and run like this:

docker build -t pip-audit .
docker run -v $PWD:/app -e PIP_AUDIT_LOGLEVEL=debug pip-audit -r requirements_for_safety.txt

With -r it is very slow (never finishes), without -r it finishes, but it isn’t a scenario I care about- if I have to install malicious code before I can audit it, what good is that, eh?

Expected behavior

It is very fast.

[woodruffw]

Nope, you understood correctly! That’s a problem with what I’m proposing, and I don’t have a good answer to it yet.

Braindump follows:

The big problem with our current approach is that we’re essentially running resolvelib N + 1 times: one at the virtualenv level each time we attempt an sdist resolution step (via pip install), and then one final time at the “top” as we collect the entire dependency graph.

Composing them in this way is unsound, since resolvelib is tasked with making global (concretizing) dependency decisions at each individual step while the “top” resolver has its own selection requirements that can’t be simultaneously honored. The end result is #197.

I think there are two ways we can fix this:

1. We can carry more information in our Resolver implementation, enabling it to backtrack further. Using #197 as an example, the trick might be keeping two sets of resolution information: the concrete versions at each step, and the (narrowed?) constraints at each step. Then, instead of our “top” resolvelib seeing two separate concrete versions of e.g. setuptools, it could instead check the compatibility of the constraints and select the maximal version that satisfies both. I’m not 100% sure what that looks like yet, partially because I don’t fully understand the different knobs that resolvelib offers 😅
2. We can give up on using resolvelib entirely, and go with a “naive” approach where we just create a new venv and forward the requirement files into it, collecting the fully resolved dependencies at the very end. We could make this roughly as responsive as the current approach by using the communicate() APIs for subprocesses, updating our AuditState each time we receive a line from the underlying pip install process.

[woodruffw]

Another incremental improvement here: #194 will make pip-audit -r about 10% faster on contrived benchmarks, and probably even faster on real workloads.