Goal: ensure lock file consistency after all installation commands

As noted in https://github.com/pypa/pipenv/issues/1137, pipenv doesn’t currently check for requirement conflicts between [packages] and [dev-packages], which can cause problems when attempting to generate a flat requirements.txt that covers both, as well as causing general weirdness when dependencies conflict.

The way pipenv install works (running the installation and then updating the lock file) means it is also relatively easy for the local environment to get out of sync with the lock file: if the installation request covers an already installed package, then it won’t be updated locally, but the lock file will be updated to the latest version available from any configured package indices.

This issue doesn’t cover an individual feature request. Instead, it establishes the goal of helping to ensure consistency between the lock file and the local environment by structuring installation commands to modify the lock file first, and then use the updated lock file to drive the actual installation step. (We don’t have any kind of target ETA for how long this work will take - writing it down is just intended to help ensure we’re all heading in the same direction, and are all comfortable with that direction)

A key aspect of this is going to be clarifying the division of responsibilities between pipenv install, pipenv uninstall, pipenv lock, and pipenv update (and potentially adding one or more new subcommands if deemed necessary).

Proposed substeps:

• add a pipenv sync subcommand that ensures that the current environment matches the lock file (akin to pip-sync in pip-tools). (Implemented by @kennethreitz for 10.0.0, with pipenv sync ensuring versions of locked packages match the lock file, while pipenv clean removes packages that aren’t in the lock file. I like that change relative to pip-sync, as it means that including an implicit sync in another command won’t unexpectedly delete anything from the virtual environment)
• Keep the current behaviour for pipenv install with no arguments, but switch to recommending the use of pipenv sync when setting up a fresh environment (that way pipenv install is used mainly for changing the installed components)
• update pipenv lock to always ensure that [packages] and [dev-packages] are consistent with each other (resolution for #1137)
• change pipenv update to be equivalent to pipenv lock && pipenv sync (pipenv update is instead being removed entirely - it’s old behaviour was actually comparable to what is now pipenv sync && pipenv clean)
• add a new pipenv lock --keep-outdated option that still generates a fresh Pipfile.lock from Pipfile, but minimises the changes made to only those needed to satisfy any changes made to Pipfile (whether that’s package additions, removals, or changes to version constraints)
• change pipenv install <packages> to be equivalent to “add or update entries in Pipfile” followed by pipenv lock && pipenv sync (implemented in #1486).
• Add a --keep-outdated option to pipenv install that it passes through to the pipenv lock operation.
• add a new pipenv install --selective-upgrade <packages> feature that’s semantically equivalent to “remove those package entries from Pipfile (if present)”, run pipenv lock --keep-outdated, then run pipenv install --keep-outdated <packages> (this is the final step that delivers support for https://github.com/pypa/pipenv/issues/966). If just a package name is given, then the existing version constraint in Pipfile is used, otherwise the given version constraint overwrites the existing one. The effect of this on the current environment should be the same as pip install --upgrade --upgrade-strategy=only-if-needed <packages> in pip 9+ (except that Pipfile.lock will also be updated).

If anyone wants to pick up one of these substeps, make a comment below to say you’re working on it (to attempt to minimise duplication of effort), then file a PR linking back to this issue. (Steps listed earlier will need to be completed before later steps are practical)

Open questions:

• None

Resolved questions (at least for the immediate future):

• pipenv lock will continue to default to upgrading everything by default, with pipenv lock --keep-outdated to request a minimal update that only adjusts the lock file to account for Pipfile changes (additions, removals, and changes to version constraints)
• pipenv lock --skip-lock will continue to work as it does today (even though it means the lock file and the local environment can get out of sync: use pipenv sync && pipenv clean to resync them)
• pipenv install and pipenv install <package> will continue to imply a full pipenv lock by default, with pipenv install --keep-outdated needed to request only the minimal changes required to satisfy the installation request
• pipenv install <package> will continue to retain the existing version constraint in Pipfile if none is given on the command line, even for the new --selective-upgrade option
• pipenv uninstall <package> will just remove the specified package[s], and hence may leave no longer needed dependencies in the local environment. Running pipenv lock && pipenv sync && pipenv clean will clear them out.

---

Note: the original proposal here was just to ensure that [dev-packages] and [packages] were kept in sync when generating the lock file, and the first few posts reflect that. The current proposal instead covers the lock file driven symmetric update proposal first mentioned in https://github.com/pypa/pipenv/issues/1255#issuecomment-354585775

[ncoghlan]

@taion By design, pipenv focuses almost entirely on the “moving target” security model I described in my LCA talk. Thus, the two intended usage models are as follows:

• Interactive batch updates:

  • A human periodically performs some action (explicit batch update, adding a new dependency, upgrading an explicit dependency)
  • As part of these operations, the lock file and current environment are updated to the latest version of everything that satisfies the expressed constraints in Pipfile
  • Over time, either the expressed constraints are updated (and chosen dependencies are switched out) until this way of working becomes painless, or else the developers switch to the second option below

• Automated selective updates:

  • An automated process keeps watch for new versions of dependencies and submits an issue+PR whenever they become available
  • Humans mainly set up new environments, explicitly add dependencies, and adjust the expresssed constraints on existing dependencies

Now, there are currently CLI design & implementation problems affecting both of those usage models - for interactive batch updates, there are ways for the lock file and the local environment to get out of sync, and for the automated selective updates, there are challenges in implementing the tools that do the selective updates, as well as in enabling the related interactive selective operations to add new dependencies and modify the constraints on existing ones.

But that core design philosophy of “Default to running the latest version of everything unless explicitly told otherwise in Pipfile” isn’t going to change - we just want to add enough selective update support to better handle the case where most lock file updates are submitted by an update management service, rather than being submitted by humans.

[kennethreitz]

--selective-upgrade is done.