Poetry selects wrong Python version if Poetry's installation differs from `python3` (or `python3` changes)

• Poetry version: 1.2.2
• Python version: CPython 3.9.12
• OS version and name: MacOS (darwin) 12.6.1 Monterey (ARM architecture)
• pyproject.toml: See below

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate. Related but not the same as far as I can tell: #7075 #6978 #6893
• I have consulted the FAQ and blog for any relevant entries or release notes.
• (N/A) If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

If Poetry’s internal virtual environment (used to execute Poetry) doesn’t match the version resolved by the python3 command, then Poetry doesn’t select the correct Python version (defined in pyproject.toml) for installation when creating a new virtual environment for a project.

There are two different ways that I’ve found to achieve this scenario.

Scenario 1:

1. Install Python 3.10 (or 3.9 or 3.8, just referring to 3.10 from now on) from https://python.org (using standard installer)
2. (Now $PATH resolves python3 to 3.10)
3. Run curl -sSL https://install.python-poetry.org | python3 - as officially supported installation mechanism
4. Install Python 3.11 from https://python.org
5. (Now $PATH resolves python3 to 3.11)

Scenario 2:

1. Install Python 3.9 (or any version other than 3.11) from https://python.org (using standard installer)
2. Install Python 3.11 from https://python.org
3. (Now $PATH resolves python3 to 3.11)
4. Run curl -sSL https://install.python-poetry.org | python3.9 -

Outcome

Relaunch the terminal to ensure $PATH changes take effect.

$ poetry install --no-root
The currently activated Python version 3.9.12 is not supported by the project (~3.11).
Trying to find and use a compatible version. 
Using python3 (3.11.0)
Creating virtualenv poetrytest in /Users/scott/Documents/automation/poetrytest/.venv
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing attrs (22.1.0)

This all looks correct, but then I activate the virtual environment with source .venv/bin/activate.

Expected

The shell prompt shows (poetrytest-py3.11) and python --version shows 3.11.0. I would also expect poetry shell to work in place of source .venv/bin/activate.

Actual

The shell prompt shows (poetrytest-py3.11) and python --version shows 3.9.12 (or whatever older version I used).

$ poetry shell

Current Python version (3.9.12) is not allowed by the project (~3.11).
Please change python executable via the "env use" command.

I’ve had a colleague verify this behaviour on their MacOS machine as well, and another colleague indicated that it may work properly on Ubuntu (but I have not verified that first hand).

Files

poetry.toml

[virtualenvs]
in-project = true

pyproject.toml (minimal packages just to show installation)

[tool.poetry]
name = "poetrytest"
version = "0.1.0"
description = ""
authors = ["Your Name <[e-mail]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
attrs = "^22.1.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock

[[package]]
name = "attrs"
version = "22.1.0"
description = "Classes Without Boilerplate"
category = "main"
optional = false
python-versions = ">=3.5"

[package.extras]
dev = ["cloudpickle", "coverage[toml] (>=5.0.2)", "furo", "hypothesis", "mypy (>=0.900,!=0.940)", "pre-commit", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "sphinx", "sphinx-notfound-page", "zope.interface"]
docs = ["furo", "sphinx", "sphinx-notfound-page", "zope.interface"]
tests = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "zope.interface"]
tests-no-zope = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins"]

[metadata]
lock-version = "1.1"
python-versions = "~3.11"
content-hash = "088f78f8dc08e7c9c2577c94e3669da3a7f2e3a6a2c4caa83caa754e5c626bef"

[metadata.files]
attrs = [
    {file = "attrs-22.1.0-py2.py3-none-any.whl", hash = "sha256:86efa402f67bf2df34f51a335487cf46b1ec130d02b8d39fd248abfd30da551c"},
    {file = "attrs-22.1.0.tar.gz", hash = "sha256:29adc2665447e5191d0e7c568fde78b21f9672d344281d0c6e1ab085429b22b6"},
]

[finswimmer]

Thanks a lot for your excellent reproducer @ShadowLNC. 👍 I can reproduce it with your instructions.

Hopefully I have some time the next days to find out what’s going on.

[ShadowLNC]

Edited to add: Since Poetry 1.3.0 has been released, these Ubuntu results are on Poetry 1.3.0.

Hi - I think I have managed to get a set of reproduction steps in docker. Ubuntu locks the python3 symlink to 3.10, so I’ve changed it such that we install Poetry against 3.9, and request 3.10 in pyproject.toml.

I created poetry-repro-setup.sh:

apt-get update -y
# Need software-properties-common for apt-add-repository
apt-get install -y software-properties-common python3.10 python3.10-venv python3.11 python3.11-venv curl
add-apt-repository -y ppa:deadsnakes/ppa
apt-get install -y python3.9 python3.9-venv  # From deadsnakes

mkdir poetrytest
cd poetrytest

cat - > poetry.toml <<EOF
[virtualenvs]
in-project = true
EOF

cat - > pyproject.toml <<EOF
[tool.poetry]
name = "poetrytest"
version = "0.1.0"
description = ""
authors = ["Your Name <[e-mail]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"
attrs = "^22.1.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
EOF

cat - > poetry.lock <<EOF
[[package]]
name = "attrs"
version = "22.1.0"
description = "Classes Without Boilerplate"
category = "main"
optional = false
python-versions = ">=3.5"

[package.extras]
dev = ["cloudpickle", "coverage[toml] (>=5.0.2)", "furo", "hypothesis", "mypy (>=0.900,!=0.940)", "pre-commit", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "sphinx", "sphinx-notfound-page", "zope.interface"]
docs = ["furo", "sphinx", "sphinx-notfound-page", "zope.interface"]
tests = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "zope.interface"]
tests-no-zope = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins"]

[metadata]
lock-version = "1.1"
python-versions = "~3.10"
content-hash = "f7ca16790b2695440ca3017a7bb30137759c6612e03aec49ae380742bbc484ed"

[metadata.files]
attrs = [
    {file = "attrs-22.1.0-py2.py3-none-any.whl", hash = "sha256:86efa402f67bf2df34f51a335487cf46b1ec130d02b8d39fd248abfd30da551c"},
    {file = "attrs-22.1.0.tar.gz", hash = "sha256:29adc2665447e5191d0e7c568fde78b21f9672d344281d0c6e1ab085429b22b6"},
]
EOF

I ran the docker container as follows:

docker run --rm -it -v $(pwd)/poetry-repro-setup.sh:/poetry-repro-setup.sh ubuntu bash

Then, within the docker container:

chmod +x poetry-repro-setup.sh
/poetry-repro-setup.sh
curl -sSL https://install.python-poetry.org | python3.9 - 
export PATH="/root/.local/bin:$PATH"
cd /poetrytest

And then when I try to install the virtual environment, I get the same behaviour as MacOS:

root@84b3855a09a8:/poetrytest# poetry install --no-root
The currently activated Python version 3.9.16 is not supported by the project (~3.10).
Trying to find and use a compatible version. 
Using python3 (3.10.6)
Creating virtualenv poetrytest in /poetrytest/.venv
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing attrs (22.1.0)
root@84b3855a09a8:/poetrytest# poetry shell

Current Python version (3.9.16) is not allowed by the project (~3.10).
Please change python executable via the "env use" command.
root@84b3855a09a8:/poetrytest# source .venv/bin/activate
(poetrytest-py3.10) root@84b3855a09a8:/poetrytest# python --version
Python 3.9.16

If, instead of installing Poetry with 3.9, I install it with 3.11 (as provided in the container via above script - an older 3.11.0rc1 version from the official Ubuntu Jammy repository, instead of deadsnakes), I get different behaviour.

• If 3.10 (pointed at by the python3 symlink) is requested by pyproject.toml and poetry.lock, it warns that 3.11 is active, finds 3.10 (Using python3 (3.10.6) is output), and installs correctly.
• If 3.11 is requested by pyproject.toml and poetry.lock (with the content hash from my original post, 088f78f8dc08e7c9c2577c94e3669da3a7f2e3a6a2c4caa83caa754e5c626bef), it does not warn (as 3.11 is active and I am requesting 3.11), but installs 3.10 instead, with the 3.11 venv prompt.

Here’s my full results (where python3 always refers to python3.10 on Ubuntu). Edited to add: Since Poetry 1.3.0 has been released, these Ubuntu results are on Poetry 1.3.0.

Poetry Installed With	Requested Python	Virtualenv Prompt	Installed Python
3.9	3.9	3.9	✅ 3.9
3.9	3.10	3.10	❌ 3.9
3.9	3.11	3.11	✅ 3.11
3.10	3.9	3.9	✅ 3.9
3.10	3.10	3.10	✅ 3.10
3.10	3.11	3.11	✅ 3.11
3.11	3.9	3.9	✅ 3.9
3.11	3.10	3.10	✅ 3.10
3.11	3.11	3.11	❌ 3.10

(For completeness, the poetry.lock content hash for Python 3.9 is 1e20473c3d8816fdca442b8208dd4268efad46603021991e7219def665cf6772.)