pip-audit not auditing the same package versions as pip installs.

Bug description

My understanding of pip-audit is that, when given a requirements.txt file, it would audit the same versions of the same packages that pip would install. This doesn’t seem to be the case.

I have a requirements.txt file containing just the cryptography module with no version constraint specified.

If I create a venv and install the package then it installs:

• cryptography 36.0.2
• cffi 1.15.0 (as cryptography has a requirement for cffi>=1.1.2)
• pcparser 2.21

If I run pip-audit -r requirement.txt then it tries to install cffi version 1.0.2-2 rather than 1.15.0 and fails with:

pip_audit._virtual_env.VirtualEnvError: Failed to install packages: ['/tmp/tmpffo5omkp/bin/python3', '-m', 'pip', 'install', '/tmp/tmp4dbeewpp/cffi-1.0.2-2.tar.gz']

If I change the requirements.txt file to contain cryptography==36.0.2, the same thing happens.

Likewise, if I append cffi==1.15.0 to the requirements.txt file pip-audit still tries, and fails, to use cffi 1.0.2-2

Reproduction steps

Running in Ubuntu 22.04 Docker container with just python3, venv, git installed. Running as a non-root user in the container I install pip-audit, either via pip, or directly from the git repo Create requirements.txt containing just cryptography Create a venv, install cryptography package from requirements.txt using pip Run pip list to confirm package versions installed as listed above (specifically cffi 1.15.0) Run pip-audit -r requirements.txt pip-audit tries to install cffi 1.0.2.2 and fails, as above

Expected behavior

pip-audit should audit the same packages and the same versions of the packages as pip install installs pip-audit should not fail

Screenshots and logs

Dockerfile used:

# Start with up-to-date Ubuntu

FROM ubuntu:22.04

# User to run with

ARG BUILD_USER=build
ARG BUILD_UID=1000
ARG BUILD_GID=1000
ARG HOME_DIR=/home/build

# Update & upgrade, install minimal Python setup

RUN apt update
RUN apt upgrade --yes
RUN apt install --yes apt-utils
RUN apt install --yes python3 python3-pip python3-venv git

# Add the user

RUN adduser --disabled-password --gecos '' ${BUILD_USER}
USER ${BUILD_UID}

WORKDIR ${HOME_DIR}

# Install pip-audit

#RUN pip install --upgrade --no-warn-script-location pip-audit
RUN python3 -m pip install git+https://github.com/trailofbits/pip-audit

Contents of requirements.txt file:

cryptography

Output from pip list in venv after installing cryptography package:

Package      Version
------------ -------
cffi         1.15.0
cryptography 36.0.2
pip          22.0.2
pycparser    2.21
setuptools   59.6.0

Output from pip-audit -v -r requirements.txt:

DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple', extra_index_urls=[], skip_editable=False)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cryptography" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cryptography HTTP/1.1" 301 118
DEBUG:cachecontrol.controller:Updating cache with response from "https://pypi.org/simple/cryptography"
DEBUG:cachecontrol.controller:Caching permanent redirect
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cryptography/" in the cache
DEBUG:cachecontrol.controller:Current age based on date: 153
DEBUG:cachecontrol.controller:Freshness lifetime from max-age: 600
DEBUG:cachecontrol.controller:The response is "fresh", returning cached response
DEBUG:cachecontrol.controller:600 > 153
DEBUG:cachecontrol.controller:Looking up "https://files.pythonhosted.org/packages/5d/a9/b73a5d6f50a7b2f6ef65a2d2a14e848b62dfc79d10d29277586a94cf1f23/cryptography-36.0.2-cp36-abi3-macosx_10_10_universal2.whl" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/5d/a9/b73a5d6f50a7b2f6ef65a2d2a14e848b62dfc79d10d29277586a94cf1f23/cryptography-36.0.2-cp36-abi3-macosx_10_10_universal2.whl HTTP/1.1" 200 4716191
DEBUG:cachecontrol.controller:Ignoring unknown cache-control directive: immutable
DEBUG:cachecontrol.controller:Updating cache with response from "https://files.pythonhosted.org/packages/5d/a9/b73a5d6f50a7b2f6ef65a2d2a14e848b62dfc79d10d29277586a94cf1f23/cryptography-36.0.2-cp36-abi3-macosx_10_10_universal2.whl"
DEBUG:cachecontrol.controller:etag object cached for 1209600 seconds
DEBUG:cachecontrol.controller:Caching due to etag
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cffi" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cffi HTTP/1.1" 301 110
DEBUG:cachecontrol.controller:Updating cache with response from "https://pypi.org/simple/cffi"
DEBUG:cachecontrol.controller:Caching permanent redirect
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cffi/" in the cache
DEBUG:cachecontrol.controller:Current age based on date: 153
DEBUG:cachecontrol.controller:Freshness lifetime from max-age: 600
DEBUG:cachecontrol.controller:The response is "fresh", returning cached response
DEBUG:cachecontrol.controller:600 > 153
DEBUG:cachecontrol.controller:Looking up "https://files.pythonhosted.org/packages/ef/23/c6f7003ebb7b4b3fe4872f112b18ee181a3ec2b137e964093a8b35d4a5bd/cffi-1.0.2-2.tar.gz" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/ef/23/c6f7003ebb7b4b3fe4872f112b18ee181a3ec2b137e964093a8b35d4a5bd/cffi-1.0.2-2.tar.gz HTTP/1.1" 200 317417
DEBUG:cachecontrol.controller:Ignoring unknown cache-control directive: immutable
DEBUG:cachecontrol.controller:Updating cache with response from "https://files.pythonhosted.org/packages/ef/23/c6f7003ebb7b4b3fe4872f112b18ee181a3ec2b137e964093a8b35d4a5bd/cffi-1.0.2-2.tar.gz"
DEBUG:cachecontrol.controller:etag object cached for 1209600 seconds
DEBUG:cachecontrol.controller:Caching due to etag
Traceback (most recent call last):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_virtual_env.py", line 103, in post_setup
    run(package_install_cmd, state=self._state)
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_subprocess.py", line 51, in run
    raise CalledProcessError(f"{pretty_args} exited with {process.returncode}")
pip_audit._subprocess.CalledProcessError: python3 -m pip install /tmp/tmps9lu0wkz/cffi-1.0.2-2.tar.gz exited with 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/build/.local/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_cli.py", line 357, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_audit.py", line 66, in audit
    for dep, vulns in self._service.query_all(specs):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_service/interface.py", line 142, in query_all
    for spec in specs:
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/requirement.py", line 98, in collect
    for _, deps in self._resolver.resolve_all(iter(req_values)):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/interface.py", line 87, in resolve_all
    yield (req, self.resolve(req))
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 73, in resolve
    result = self.resolver.resolve([req])
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 341, in get_dependencies
    return candidate.dependencies
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 119, in dependencies
    self._dependencies = list(self._get_dependencies())
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 101, in _get_dependencies
    deps: List[str] = self.metadata.get_all("Requires-Dist", [])
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 94, in metadata
    self._metadata = self._get_metadata_for_sdist()
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 162, in _get_metadata_for_sdist
    ve.create(ve_dir)
  File "/usr/lib/python3.10/venv/__init__.py", line 78, in create
    self.post_setup(context)
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_virtual_env.py", line 105, in post_setup
    raise VirtualEnvError(f"Failed to install packages: {package_install_cmd}") from cpe
pip_audit._virtual_env.VirtualEnvError: Failed to install packages: ['/tmp/tmpjk63y53b/bin/python3', '-m', 'pip', 'install', '/tmp/tmps9lu0wkz/cffi-1.0.2-2.tar.gz']

Platform information

• OS name and version: Ubuntu 22.04 (in Docker container)
• pip-audit version (pip-audit -V): pip-audit 2.1.0
• Python version (python -V or python3 -V): Python 3.10.3
• pip version (pip -V or pip3 -V): pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)

[woodruffw]

Upstream: https://github.com/pypa/packaging/issues/527

[woodruffw]

Cut with 2.1.1; should be available on PyPI shortly. Thanks again for reporting!