pipenv 2022.8.5.13 introduced an issue with platformdirs

Issue description

We have a job in our CI pipeline to uses pipenv to set up an environment with black and then run said tool. Our CI job for this in Gitlab-CI is extremely simple:

black_linting_check:
  stage: lint
  image: python:3.8
  before_script:
    - python -V
    - pip install pipenv
    - pipenv install --dev
  script:
    - pipenv run black derms derms_core mocks --check

Unfortunately it seems like the release of pipenv 2022.8.5.13 introduced some issue with the platformdirs package.

The following is a log of a CI successful run of our job from August 12th 2022:

Running with gitlab-runner 14.7.0 (98daeee0)
  on ip-17[2](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L2)-[3](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L3)1-5-168 qLr1uLbu
Resolving secrets
00:00
Preparing the "docker" executor
00:09
Using Docker executor with image python:3.8 ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image python:3.8 ...
Using docker image sha2[5](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L5)[6](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L6):be03ed85[7](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L7)d3127[8](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L8)de24758c22[9](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L9)9c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
Preparing environment
01:12
Running on runner-qlr1ulbu-project-22339857-concurrent-1 via ip-172-31-5-168...
Getting source from Git repository
00:38
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/opusonesolutions/gridos/derms/v3/derms-core/.git/
Created fresh repository.
Checking out 197c530c as DR-2998-deadband-settings-api...
Skipping Git submodules setup
Executing "step_script" stage of the job script
04:06
Using docker image sha256:be03ed857d31278de24758c2299c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
$ python -V
Python 3.8.13
$ pip install pipenv
Looking in indexes: https://pypi.org/simple, https://[MASKED]:****@opusonesolutions.jfrog.io/opusonesolutions/api/pypi/pypi-local/simple
Collecting pipenv
  Downloading pipenv-2022.8.5-py2.py3-none-any.whl (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 64.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.8/site-packages (from pipenv) (57.5.0)
Collecting certifi
  Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 27.6 MB/s eta 0:00:00
Collecting virtualenv
  Downloading virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 63.1 MB/s eta 0:00:00
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Collecting distlib<1,>=0.3.5
  Downloading distlib-0.3.5-py2.py3-none-any.whl (466 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.0/467.0 KB 59.8 MB/s eta 0:00:00
Collecting platformdirs<3,>=2.4
  Downloading platformdirs-2.5.2-py3-none-any.whl ([14](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L14) kB)
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: distlib, virtualenv-clone, platformdirs, filelock, certifi, virtualenv, pipenv
Successfully installed certifi-2022.6.[15](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L15) distlib-0.3.5 filelock-3.8.0 pipenv-2022.8.5 platformdirs-2.5.2 virtualenv-20.[16](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L16).3 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$ pipenv install --dev
Creating a virtualenv for this project...
Pipfile: /builds/opusonesolutions/gridos/derms/v3/derms-core/Pipfile
Using /usr/local/bin/python3.8 (3.8.13) to create virtualenv...
created virtual environment CPython3.8.13.final.0-64 in 6[19](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L19)ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/derms-core-Sx_JEpdN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==[22](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L22).2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/derms-core-Sx_JEpdN
Installing dependencies from Pipfile.lock ([29](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L29)83d4)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
$ pipenv run black derms derms_api mocks --check
Loading .env environment variables...
All done! ✨ 🍰 ✨
290 files would be left unchanged.
Cleaning up project directory and file based variables
00:01
Job succeeded

The following is a log of a CI failed run of our job from August 13th 2022:

Running with gitlab-runner 14.7.0 (98daeee0)
  on ip-17[2](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L2)-[3](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L3)1-5-63 oyxCD8yZ
Resolving secrets
00:00
Preparing the "docker" executor
01:2[5](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L5)
Using Docker executor with image python:3.8 ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image python:3.8 ...
Using docker image sha25[6](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L6):be03ed85[7](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L7)d3127[8](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L8)de24758c22[9](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L9)9c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
Preparing environment
00:03
Running on runner-oyxcd8yz-project-22339857-concurrent-1 via ip-172-31-5-63...
Getting source from Git repository
00:11
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/opusonesolutions/gridos/derms/v3/derms-core/.git/
Checking out ca798f52 as DR-2998-deadband-settings-api...
Skipping Git submodules setup
Executing "step_script" stage of the job script
02:49
Using docker image sha256:be03ed857d31278de24758c2299c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
$ python -V
Python 3.8.13
$ pip install pipenv
Looking in indexes: https://pypi.org/simple, https://[MASKED]:****@opusonesolutions.jfrog.io/opusonesolutions/api/pypi/pypi-local/simple
Collecting pipenv
  Downloading pipenv-2022.8.13-py2.py3-none-any.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 16.2 MB/s eta 0:00:00
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Collecting certifi
  Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 2.4 MB/s eta 0:00:00
Collecting virtualenv
  Downloading virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB [12](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L12).9 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.8/site-packages (from pipenv) (57.5.0)
Collecting platformdirs<3,>=2.4
  Downloading platformdirs-2.5.2-py3-none-any.whl ([14](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L14) kB)
Collecting distlib<1,>=0.3.5
  Downloading distlib-0.3.5-py2.py3-none-any.whl (466 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.0/467.0 KB 3.7 MB/s eta 0:00:00
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: distlib, virtualenv-clone, platformdirs, filelock, certifi, virtualenv, pipenv
Successfully installed certifi-2022.6.[15](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L15) distlib-0.3.5 filelock-3.8.0 pipenv-2022.8.13 platformdirs-2.5.2 virtualenv-20.[16](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L16).3 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version [22](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L22).0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$ pipenv install --dev
Creating a virtualenv for this project...
Pipfile: /builds/opusonesolutions/gridos/derms/v3/derms-core/Pipfile
Using /usr/local/bin/python3.8 (3.8.13) to create virtualenv...
created virtual environment CPython3.8.13.final.0-64 in 1679ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/derms-core-Sx_JEpdN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/derms-core-Sx_JEpdN
Installing dependencies from Pipfile.lock ([29](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L29)83d4)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
$ pipenv run black derms derms_api mocks --check
Loading .env environment variables...
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/derms-core-Sx_JEpdN/bin/black", line 5, in <module>
    from black import patched_main
  File "src/black/__init__.py", line 42, in <module>
  File "src/black/nodes.py", line [33](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L33), in <module>
  File "src/black/cache.py", line 9, in <module>
ModuleNotFoundError: No module named 'platformdirs'
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

I am pretty sure that this issue is caused by some change between pipenv 2022.8.5 and 2022.8.5.13 for the following reasons:

1. I noticed this issue initially on a different MR in our Gitlab and so went back and forcibly re-ran the job show above on a different MR that had not changed between the two job runs show above. The result was this previously green job failing with no change other than the version of pipenv (Which, you will note from the CI job details was not version locked)
2. Modifying our CI job such that the version of pipenv installed was locked to 2022.8.5 corrected the issue

Expected result

pipenv should install platformdirs correctly into the environment

Actual result

pipenv seems to fail to install platformdirs into the enviroment

Steps to replicate

This should be replicable using the steps in the CI job listed above along with any old random codebase

[matteius]

pipenv==2022.8.15 has been released.

[chruss2]

Yes, the latest commit works now. 👍

BTW, I was curious why this was happening, so I looked into how pip is vendored. The vendoring is adding line 29 to main.py: sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))))

This line is needed in order to import pipenv.patched.pip._internal.cli.main, but also affects how packages are installed. That line is not reached if you call pip with -m pip, but is reached now that you’re calling __pip-runner__.py directly, which was a change in 2022.8.13.

I figured I’d mention this, since changing sys.path may affect pip in other areas of the code too.