Goal: ensure lock file consistency after all installation commands
See original GitHub issueAs noted in https://github.com/pypa/pipenv/issues/1137, pipenv
doesn’t currently check for requirement conflicts between [packages]
and [dev-packages]
, which can cause problems when attempting to generate a flat requirements.txt
that covers both, as well as causing general weirdness when dependencies conflict.
The way pipenv install
works (running the installation and then updating the lock file) means it is also relatively easy for the local environment to get out of sync with the lock file: if the installation request covers an already installed package, then it won’t be updated locally, but the lock file will be updated to the latest version available from any configured package indices.
This issue doesn’t cover an individual feature request. Instead, it establishes the goal of helping to ensure consistency between the lock file and the local environment by structuring installation commands to modify the lock file first, and then use the updated lock file to drive the actual installation step. (We don’t have any kind of target ETA for how long this work will take - writing it down is just intended to help ensure we’re all heading in the same direction, and are all comfortable with that direction)
A key aspect of this is going to be clarifying the division of responsibilities between pipenv install
, pipenv uninstall
, pipenv lock
, and pipenv update
(and potentially adding one or more new subcommands if deemed necessary).
Proposed substeps:
- add a
pipenv sync
subcommand that ensures that the current environment matches the lock file (akin topip-sync
inpip-tools
). (Implemented by @kennethreitz for 10.0.0, withpipenv sync
ensuring versions of locked packages match the lock file, whilepipenv clean
removes packages that aren’t in the lock file. I like that change relative topip-sync
, as it means that including an implicit sync in another command won’t unexpectedly delete anything from the virtual environment) - Keep the current behaviour for
pipenv install
with no arguments, but switch to recommending the use ofpipenv sync
when setting up a fresh environment (that waypipenv install
is used mainly for changing the installed components) - update
pipenv lock
to always ensure that[packages]
and[dev-packages]
are consistent with each other (resolution for #1137) - change
pipenv update
to be equivalent topipenv lock && pipenv sync
(pipenv update is instead being removed entirely - it’s old behaviour was actually comparable to what is nowpipenv sync && pipenv clean
) - add a new
pipenv lock --keep-outdated
option that still generates a freshPipfile.lock
fromPipfile
, but minimises the changes made to only those needed to satisfy any changes made toPipfile
(whether that’s package additions, removals, or changes to version constraints) - change
pipenv install <packages>
to be equivalent to “add or update entries in Pipfile” followed bypipenv lock && pipenv sync
(implemented in #1486). - Add a
--keep-outdated
option to pipenv install that it passes through to thepipenv lock
operation. - add a new
pipenv install --selective-upgrade <packages>
feature that’s semantically equivalent to “remove those package entries from Pipfile (if present)”, runpipenv lock --keep-outdated
, then runpipenv install --keep-outdated <packages>
(this is the final step that delivers support for https://github.com/pypa/pipenv/issues/966). If just a package name is given, then the existing version constraint inPipfile
is used, otherwise the given version constraint overwrites the existing one. The effect of this on the current environment should be the same aspip install --upgrade --upgrade-strategy=only-if-needed <packages>
in pip 9+ (except thatPipfile.lock
will also be updated).
If anyone wants to pick up one of these substeps, make a comment below to say you’re working on it (to attempt to minimise duplication of effort), then file a PR linking back to this issue. (Steps listed earlier will need to be completed before later steps are practical)
Open questions:
- None
Resolved questions (at least for the immediate future):
pipenv lock
will continue to default to upgrading everything by default, withpipenv lock --keep-outdated
to request a minimal update that only adjusts the lock file to account forPipfile
changes (additions, removals, and changes to version constraints)pipenv lock --skip-lock
will continue to work as it does today (even though it means the lock file and the local environment can get out of sync: usepipenv sync && pipenv clean
to resync them)pipenv install
andpipenv install <package>
will continue to imply a fullpipenv lock
by default, withpipenv install --keep-outdated
needed to request only the minimal changes required to satisfy the installation requestpipenv install <package>
will continue to retain the existing version constraint inPipfile
if none is given on the command line, even for the new--selective-upgrade
optionpipenv uninstall <package>
will just remove the specified package[s], and hence may leave no longer needed dependencies in the local environment. Runningpipenv lock && pipenv sync && pipenv clean
will clear them out.
Note: the original proposal here was just to ensure that [dev-packages]
and [packages]
were kept in sync when generating the lock file, and the first few posts reflect that. The current proposal instead covers the lock file driven symmetric update proposal first mentioned in https://github.com/pypa/pipenv/issues/1255#issuecomment-354585775
Issue Analytics
- State:
- Created 6 years ago
- Comments:50 (36 by maintainers)
@taion By design,
pipenv
focuses almost entirely on the “moving target” security model I described in my LCA talk. Thus, the two intended usage models are as follows:Interactive batch updates:
Pipfile
Automated selective updates:
Now, there are currently CLI design & implementation problems affecting both of those usage models - for interactive batch updates, there are ways for the lock file and the local environment to get out of sync, and for the automated selective updates, there are challenges in implementing the tools that do the selective updates, as well as in enabling the related interactive selective operations to add new dependencies and modify the constraints on existing ones.
But that core design philosophy of “Default to running the latest version of everything unless explicitly told otherwise in
Pipfile
” isn’t going to change - we just want to add enough selective update support to better handle the case where most lock file updates are submitted by an update management service, rather than being submitted by humans.--selective-upgrade
is done.