Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

extend fast-deps to sdists (and allow downloading foreign sdists) (working prototype!)

See original GitHub issue

What’s the problem this feature will solve?

--use-feature=fast-deps is awesome. After #8896, the stage is set to try downloading all the lazy deps in parallel at once! However, we still run into a mild slowdown by downloading, unpacking, then doing some further processing on every non-wheel dependency! It turns out this isn’t necessary.
It is not possible to invoke pip to resolve transitive dependencies for pip download or for a --dry-run resolve as per #53 if any of those dependencies are only provided as sdists.

Describe the solution you’d like Extend fast-deps to sdists by extracting the single PKG-INFO file, without any further processing. This file existed in every sdist download I was able to quickly check on pypi (~5000 projects).

This should improve the parallelism available when building sdists in pip wheel, and downloading dists in pip download, once that feature is implemented.
As described in https://github.com/pypa/pip/issues/53#issuecomment-646445680, it would be wonderful to perform a “dry run” resolve without having to incur the full download costs at all. The use case described in that comment is a PEX file which resolves its dependencies when first executed, allowing the deployed PEX to be much much smaller. Right now, it would still be impossible to deploy such an “ipex” file to a foreign platform if there were any sdist-only dependencies, even if #53 was otherwise fully implemented.

Alternative Solutions Pex invokes pip for all download, wheel-building, and installation tasks, so its API for resolving wheels at least is predicated upon any limitations in pip. In this case, it would be lovely to improve download performance as well as enable dry-run resolves in Pex, but it’s not clear to me whether there is any workaround without upstream pip changes.

Additional context This is prototyped in the branch https://github.com/cosmicexplorer/pip/tree/fast-deps-sdists (on top of #8896) and appears to work really smoothly.