Pipenv searches virtual environment's python binary in wrong path

Hello,

I’m having trouble using Pipenv lately. In particular, after I updated my OS (Fedora 36), pipenv during the install command tries to search the python binary inside {local_virtualenv}/local/bin/python, while the symlink to python is in {local_virtualenv}/bin/python. I tried to remove and reinstall the virtual environment (pipenv --rm and pipenv install), but the problem persists.

Application versions

➜  etl git:(develop) pip --version
pip 22.1.1 from /home/gfurlan/.local/lib/python3.10/site-packages/pip (python 3.10)
➜  etl git:(develop) pipenv --version
pipenv, version 2022.5.2
➜  etl git:(develop) python --version
Python 3.10.4
➜  etl git:(develop) which python
/usr/bin/python

Pipenv install

➜  etl git:(develop) pipenv install -d
Creating a virtualenv for this project...
Pipfile: /home/gfurlan/src/project/glue/etl/Pipfile
Using /usr/bin/python3.7m (3.7.13) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.7.13.final.0-64 in 89ms
  creator CPython3Posix(dest=/home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/gfurlan/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Traceback (most recent call last):
  File "/home/gfurlan/.local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/cli/options.py", line 56, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/cli/command.py", line 222, in install
    do_install(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 1965, in do_install
    ensure_project(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 553, in ensure_project
    ensure_virtualenv(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 486, in ensure_virtualenv
    do_create_virtualenv(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 1013, in do_create_virtualenv
    project._environment = Environment(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/environment.py", line 70, in __init__
    self._base_paths = self.get_paths()
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/environment.py", line 394, in get_paths
    c = subprocess_run(command)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/utils/processes.py", line 75, in subprocess_run
    return subprocess.run(
  File "/usr/lib64/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib64/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/local/bin/python'

pipenv shell and echo $PATH:

➜  etl git:(develop) pipenv shell
Launching subshell in virtual environment...
 . /home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/bin/activate
➜  etl git:(develop)  . /home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/bin/activate
(etl) ➜  etl git:(develop) echo $PATH
/home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/bin:/home/gfurlan/.local/bin:/home/gfurlan/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

As you can see the PATH variable is correctly set, though pipenv tries to use the local prefix for calling python for some reason.

May you please help me sorting this out?

Thanks

EDIT: I think I got the point where the issue starts, that is in environment.py, in function script_basedir which gets the paths from sysconfig.get_paths, setting base and platbase variables to the virtual environment prefix. That function returns for the scripts key /my/custom/venv/path/local/bin.

Now the question is: if it uses sysconfig.get_paths to find python, shouldn’t it set PATH and the symlink to python accordingly, during the initial setup?

EDIT2: reading sysconfig.py from the OS installation, the following is written:

# backup the original posix_prefix as rpm_prefix
# RPM packages use it and we need to be able to read it even when changed
_INSTALL_SCHEMES['rpm_prefix'] = _INSTALL_SCHEMES['posix_prefix']
# Virtualenv >= 20.10.0 favors the "venv" scheme over the defaults when creating virtual environments.
# See: https://github.com/pypa/virtualenv/commit/8da79db86d8a5c74d03667a40e64ff832076445e
# See: https://bugs.python.org/issue45413
# "venv" should be the same as the unpatched posix_prefix for us,
# so new virtual environments aren't created with paths like venv/local/bin/python.
_INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['posix_prefix']

Thus in pipenv’s environment.py file, script_basedir function, if I overwrite the install_scheme as follows:

install_scheme = "nt" if (os.name == "nt") else "venv" # "venv" instad of "posix_prefix"

the path is now correct again.

The problem seems to be fixed now. I wonder if this is the default Python behavior, of if it’s some kind of Fedora 36 customization?

EDIT3: and I’m still unsure whether the EDIT2 sysconfig.py code is a custom Fedora add-in, but I found this issue on the CPython github: https://github.com/python/cpython/issues/89576

[matteius]

2022.6.7 has been released.

[matteius]

FWIW it has been merged into master branch of pipenv and I plan to do a pipenv release at some point soon, like say in the first week of June, probably. Trying to see what else we can get into that release that would be fixing an issue as well.