Pipenv lock changes markers in a non deterministic way

Issue description

I’m running pipenv version 2020.11.15 in a Gitlab CI pipeline. I use pipenv lock and then git diff Pipfile.lock to see if the Pipfile.lock was updated, meaning that the version on the repository is not up-to-date anymore.

Occasionally, pipenv updates the lock file just changing the Python 4 specifier as follows:

diff --git a/Pipfile.lock b/Pipfile.lock
index c8d17c6..0228d56 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -225,7 +225,7 @@
                 "sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
                 "sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
             ],
-            "markers": "python_version >= '3.6' and python_version < '4.0'",
+            "markers": "python_version >= '3.6' and python_version < '4'",
             "version": "==5.8.0"
         },
         "jedi": {
@@ -881,7 +881,7 @@
                 "sha256:2f4da4594db7e1e110a944bb1b551fdf4e6c136ad42e4234131391e21eb5b0df",
                 "sha256:e7b021f7241115872f92f43c6508082facffbd1c048e3c6e2bb9c2a157e28937"
             ],
-            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4.0'",
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'",
             "version": "==1.26.4"
         },
         "wcwidth": {

And sometimes it does the opposite thing, changing from python_version < '4' to python_version < '4.0'. This of course breaks the pipeline, because there is difference in the Pipfile.lock, even though that difference is not significant.

Expected result

pipenv lock should change the lock file in a deterministic way, either always with python_version < '4' or python_version < '4.0'.

[matteius]

Perhaps too early to tell, but I maybe am onto something about this issue – I removed the lru_cache decorators around the marker generation in requirementslib in this PR: https://github.com/pypa/pipenv/pull/5373

I updated @dqkqd example Dockerfile to install this branch, and ran it three times without issue:

matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issues-4660$ vim Dockerfile 
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issues-4660$ docker build .
Sending build context to Docker daemon  62.46kB
Step 1/7 : FROM archlinux
 ---> 546c416f6b9d
Step 2/7 : RUN pacman -Syu diffutils python3 python-pip git --noconfirm &> /dev/null
 ---> Using cache
 ---> 41664f828c96
Step 3/7 : RUN python -m pip install "git+https://github.com/pypa/pipenv.git@marker-determinism#egg=pipenv"  &> /dev/null
 ---> Running in 22c5f25c4438
Removing intermediate container 22c5f25c4438
 ---> c02c60d77ad3
Step 4/7 : WORKDIR /root
 ---> Running in d0e82db1ef6c
Removing intermediate container d0e82db1ef6c
 ---> 3333d4a56886
Step 5/7 : COPY Pipfile .
 ---> 42d116d6b71d
Step 6/7 : COPY script.sh .
 ---> f1089d4652f4
Step 7/7 : ENTRYPOINT ["./script.sh"]
 ---> Running in c0674607980b
Removing intermediate container c0674607980b
 ---> 964680f6585f
Successfully built 964680f6585f
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issues-4660$ docker run 964680f6585f
pipenv, version 2022.9.22.dev0
attemp 1: same
attemp 2: same
attemp 3: same
attemp 4: same
attemp 5: same
attemp 6: same
attemp 7: same
attemp 8: same
attemp 9: same
attemp 10: same
attemp 11: same
attemp 12: same
attemp 13: same
attemp 14: same
attemp 15: same
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issues-4660$ docker run 964680f6585f
pipenv, version 2022.9.22.dev0
attemp 1: same
attemp 2: same
attemp 3: same
attemp 4: same
attemp 5: same
attemp 6: same
attemp 7: same
attemp 8: same
attemp 9: same
attemp 10: same
attemp 11: same
attemp 12: same
attemp 13: same
attemp 14: same
attemp 15: same
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issues-4660$ docker run 964680f6585f
pipenv, version 2022.9.22.dev0
attemp 1: same
attemp 2: same
attemp 3: same
attemp 4: same
attemp 5: same
attemp 6: same
attemp 7: same
attemp 8: same
attemp 9: same
attemp 10: same
attemp 11: same
attemp 12: same
attemp 13: same
attemp 14: same
attemp 15: same
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issues-4660$ docker run 964680f6585f
pipenv, version 2022.9.22.dev0
attemp 1: same
attemp 2: same
attemp 3: same
attemp 4: same
attemp 5: same
attemp 6: same
attemp 7: same
attemp 8: same
attemp 9: same
attemp 10: same
attemp 11: same
attemp 12: same
attemp 13: same
attemp 14: same
attemp 15: same
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-issues-4660$ cat Dockerfile 
FROM archlinux

RUN pacman -Syu diffutils python3 python-pip git --noconfirm &> /dev/null
RUN python -m pip install "git+https://github.com/pypa/pipenv.git@marker-determinism#egg=pipenv"  &> /dev/null

WORKDIR /root
COPY Pipfile .
COPY script.sh .

ENTRYPOINT ["./script.sh"]

I also relocked pipenv 4 times and the markers only changed the first time suggesting that the branch that removes lru_cache makes it determinisitic. 🤞

[matteius]

@matpompili and @jshwi please see: https://github.com/pypa/pipenv/issues/4967 TLDR: Pip 22.0.4 upgrade branch seems to make this deterministic again.

Closing this ticket now since we are already tracking it in the newer ticket.