poetry add/install not using venv python

Ran into a bit of a python environment error when trying to install unruly packages (where they explicitly import dependency modules in their setup.py to see if it exists).

The current poetry environment package install process uses the python executable of whatever is running poetry and never actually the python of the venv itself.

https://github.com/python-poetry/poetry/blob/bf30ca696aa47bcc42937910d620c95c95c0741e/poetry/inspection/info.py#L453-L456

Because here, if no executable argument is provided to build_venv(), it defaults to the executable running poetry—could be anything anywhere with any packages!—(via executable or sys.executable). If we are using an explicit env set by poetry env use which isn’t the same binary running poetry itself, the install process will never have a usable environment we expect:

https://github.com/python-poetry/poetry/blob/bf30ca696aa47bcc42937910d620c95c95c0741e/poetry/utils/env.py#L804-L819

Proper suggested fix: use the current value of the project’s env python setting for all installs. (left as an exercise for the reader)

Hack fix: change the executable discovery line to at least consult the environment for python before defaulting to the current executable: executable or shutil.which("python3") or sys.executable, so it will at least be able to install when attempted inside the project’s own poetry shell session.

[sinoroc]

@mattsta If I were you, I think maybe I would look into deploying a private local index (i.e. a PyPI alternative like devpi) and upload my own pre built wheels of those “difficult” projects (detectron2, pytorch, etc.) on it.

[finswimmer]

but but but… using the project environment as the first build step seems reasonable, right? If there’s no difference, then what’s the harm in using the project’s environment for both build stages?

Building an external package should never happen within the current environment. Building the package can introduce new dependencies only for the build step, that you don’t want to have in your environment. So whether you use the python version that’s available within the venv or outside doesn’t matter here. Because this python is only used to create a new temporary venv with no other packages available then setuptools and wheel.