pipenv lock --keep-outdated does not respect Pipfile.lock constraints

Issue description

When I have Pipfile with a non-locked dependency (*) but the dependency is locked in Pipfile.lock and then I lock dependencies with --keep-outdated flag in the current constraints latest available version of the package is used.

Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"

Pipfile.lock

[...]
        "atomicwrites": {
            "hashes": [
                "sha256:03472c30eb2c5d1ba9227e4c2ca66ab8287fbfbbda3888aa93dc2e28fc6811b4",
                "sha256:75a9445bac02d8d058d5e1fe689654ba5a6556a1dfd8ce6ec55a0ed79866cfa6"
            ],
            "version": "==1.3.0"
        }
        "pytest": {
            "hashes": [
                "sha256:27abc3fef618a01bebb1f0d6d303d2816a99aa87a5968ebc32fe971be91eb1e6",
                "sha256:58cee9e09242937e136dbb3dab466116ba20d6b7828c7620f23947f37eb4dae4"
            ],
            "index": "pypi",
            "version": "==5.2.2"
        }
[...]

verbose output

Current constraints:
  pytest

Expected result

verbose output

Current constraints:
  pytest==5.2.2

In the case of pytest it results in not installing proper sub-dependencies. Version 5.3 dropped the requirement for atomicwrites, so the newly generated Pipfile.lock does not contain it, but since version of pytest which I actually use is 5.2.2 I’m getting an ImportError

[januszm]

@matteius

I noticed you thumbs downed my last comment, but its the truth.

I believe this is more of your own opinion rather than universal truth. I noticed that you and probably other pipenv maintainers have a fundamentally different understanding of how a dependency management system should work when it comes to package installation. Since there are so many differences here, it seems that the Python community needs another system as an alternative to pipenv. It’s a pity, because it’s always more effective to focus all efforts and budget on improving one solution, as was the case with bundler for Ruby. Perhaps poetry will be developed in the direction expected by the other part of the community that has spoken on this and similar issues. E.g. see https://github.com/pypa/pipenv/issues/3150#issuecomment-522947210 as this comment sums up the expectations of “the other part” of the community.

I think I’ll end my participation in this discussion here because neither I nor others present here (and in related issues) will convince you to change the design decisions regarding the installation, nor, as you have probably noticed, you will not convince others.

[matteius]

Also the fact that pipenv install alters other, unrelated package dependencies sounds like “feature envy”.

The other unrelated packages are actually related to your Pipfile and the specifiers you install there. All dependencies may affect the resolution of the other dependencies and so a complete resolver phase has to happen and this is not just my opinion, without this, you get the kind of behavior that --keep-outdated provides which leads to plenty of issue reports because the resolver results were not respected. The only input into the resolver is the Pipfile and its the end user’s responsibility to pick reasonable specifiers for their top level packages and things they care to pin.

even if I specify pymssql = “~=2.2” and thenpipenv install numpy it may upgrade pymssql , which I then have to test separately and what’s more, if by any chance pymssql minor upgrade introduces a “bug”, which may always happen, it will make debugging harder (it could be avoided in the first place).

it could be avoided in the first place by recognizing this fact (which you have) and then pinning it in your Pipfile until you are ready to put the time into testing the upgrade of pymssql.

A flawed implementation does not mean the whole concept is wrong. At least pipenv should have a sane option of keeping the unrelated packages untouched.

This is not how the pip resolver needs to work. you think the packages are unrelated, but they are related to the other packages you are resolving/installing and those packages apparently did not restrict it in the way you want, so please use the Pipfile for this purpose.

The common problem theme here with --keep-outdated is that “I wanted pipenv to do what I want it to, but I don’t want to tell it how.” There is no magic forumla to figure out that you want your pymssql pinned, but that you want all of your dependencies to all resolve and install correctly, so you have to be explicit and not look for a magic bullet like --keep-outdated. We are likely to deprecate this flag in the near future and remove it after that.