Make pipenv sync skip installed dependencies

We are currently using pipenv sync to deploy some of our apps with the versions of packages that we have tested. The problem we’re having is just how long it takes for a sync to run. We might install just one new package (or update one version), then have to wait several minutes for a sync to complete.

Using pipenv sync --verbose it looks like a pip install is being called for every single dependency. Wouldn’t it be faster to check the existing hashes in the lock file against the dependencies which are already installed to confirm that they match? I don’t actually know what those hashes are calculated over so this is a bit of a naive request. Ultimately my goal is just to make pipenv sync not spin for several minutes on existing dependencies when they don’t need to be updated.

Also of note, we came from using our own virtual environments with a normal requirements file for pip. Adding a new dependency or updating a version was much quicker then, so it has to be something specific to the way pipenv is running independent installs that takes so long.

Currently using Python 3.6.6 with pip 18.1 and pipenv 2018.10.13

[ncoghlan]

Edit: if running pipenv 2020.5.28 or later, pipenv sync should do the right thing by default, and the below workaround should no longer be needed.

---

When an environment is already up to date, I’d expect pipenv sync --dev to be at least as fast as combining it with pip-tools and running:

pipenv lock --keep-outdated --requirements > requirements.txt
pipenv run pip install -c requirements.txt pip-tools
pipenv run pip-sync 

At the moment, this isn’t the case: the running time of pipenv sync --dev depends on the number of packages in the environment, rather than the number of packages that have changed version since the last sync operation.

Does it really make sense for every pipenv sync call to be verifying the environment against the wheel archives? If I wanted that behaviour, I’d expect to have to pass a --verify-installed flag or similar, rather than having it be the default.

Note: to check hashes properly, at least the first install should still be done with a regular pipenv sync call (the above faster sync status checking operation doesn’t check the hashes if it actually does need to install something)

Edits:

• updated the example to match what we’re actually doing to speed up our sync processes (without the second line to ensure pip-tools has been installed, the pip-sync line would fail in a fresh venv)
• removed --dev, as that doesn’t work as expected due to #3316
• added note about the lack of hashes in the generated requirements.txt file (as per #4189)

[ncoghlan]

Note: anyone using my suggested workaround above will need to be aware of #3316, as passing --dev when generating a requirements file means you may end up with a requirements.txt file that’s missing production dependencies that you expected to be included.