Pip-audit does not deduplicate identical packages in multiple requirements files

Bug description

(venv) $ pip-audit -s osv -r requirements/base.txt -r requirements/test.txt
Found 2 known vulnerabilities in 2 packages 
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------
django-celery-results 2.2.0   GHSA-fvx8-v524-8579

Reproduction steps

1. Add a vulnerable version of some package to a “base.txt” requirements file.
2. Add the same package version to a separate “test.txt” requirements file.
3. Audit both the “base.txt” and “test.txt” requirements files at the same time using the command above.

Expected behavior

Pip-audit sees that the package names + versions are identical across both files, and reports only one vulnerability in one package. Example:

(venv) $ pip-audit -s osv -r requirements/base.txt -r requirements/test.txt
Found 1 known vulnerability in 1 package 
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------
django-celery-results 2.2.0   GHSA-fvx8-v524-8579

Screenshots and logs

Similarly, if applicable and possible, re-run the command with --verbose, and paste the logs in the code block below:

(venv) $ pip-audit -s osv -r requirements/base.txt -r requirements/test.txt --verbose
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements/base.txt' mode='r' encoding='UTF-8'>, <_io.TextIOWrapper name='requirements/test.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Osv: 'osv'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple', extra_index_urls=[], skip_editable=False, output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.osv.dev:443
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2 repeated many times
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 1360
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2 repeated many times
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 1360
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2 repeated many times
DEBUG:urllib3.connectionpool:https://api.osv.dev:443 "POST /v1/query HTTP/1.1" 200 2
Found 2 known vulnerabilities in 2 packages
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------
django-celery-results 2.2.0   GHSA-fvx8-v524-8579

Platform information

• OS name and version: RHEL 8.6
• pip-audit version (pip-audit -V): pip-audit 2.2.1
• Python version (python -V or python3 -V): Python 3.9.6
• pip version (pip -V or pip3 -V): pip 22.0.4 from my project’s venv/lib/python3.9/site-packages/pip (python 3.9)

[juspence]

Hi @tetsuo-cpp, we don’t have any transitive dependencies in our files. We use pip-tools to iterate all dependencies, get a version + hash for each, and put all dependencies (including transitive ones) into a single file.

We then have multiple files with duplicated contents, one for each set of dependencies. So base dependencies only for deployment, base + local dependencies for development / debugging, base + CI dependencies for running tests, and so on. But even a file with only one dependency has this problem, like below:

(venv) $ cat requirements/base.txt 
django-celery-results==2.3.1 \
    --hash=sha256:b8c9416619dbcc38f13398e31bcb1f14a228cd1e8f65fb22d3b7fc68aaa5331a \
    --hash=sha256:bf24ecc29c42e49cc7eb30b9b3739471331e2a0ca517cc88ca53a0cf3a2031d1

(venv) $ cat requirements/test.txt 
django-celery-results==2.3.1 \
    --hash=sha256:b8c9416619dbcc38f13398e31bcb1f14a228cd1e8f65fb22d3b7fc68aaa5331a \
    --hash=sha256:bf24ecc29c42e49cc7eb30b9b3739471331e2a0ca517cc88ca53a0cf3a2031d1

Auditing the base and test files above still shows the same issue:

(venv) $ pip-audit -s osv -r requirements/base.txt -r requirements/test.txt 
Found 2 known vulnerabilities in 2 packages
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------
django-celery-results 2.3.1   GHSA-fvx8-v524-8579

(venv) $ pip-audit -s pypi -r requirements/base.txt -r requirements/test.txt 
Found 2 known vulnerabilities in 2 packages
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------
django-celery-results 2.3.1   GHSA-fvx8-v524-8579

(venv) $ pip-audit -r requirements/base.txt -r requirements/test.txt 
Found 2 known vulnerabilities in 2 packages
Name                  Version ID                  Fix Versions
--------------------- ------- ------------------- ------------
django-celery-results 2.3.1   GHSA-fvx8-v524-8579

I tried with pip-audit==2.3.0 (what I had installed now) and with pip-audit==2.3.3 (latest).

EDIT: Maybe the issue here is that the GHSA does not specify any “Patched versions”: https://github.com/advisories/GHSA-fvx8-v524-8579

[tetsuo-cpp]

@juspence If you upgrade to v2.3.4, this should now be fixed. I’m going to go ahead and close this but please let me know if you run into any issues.