Create an implicit virtual environment based on pyproject.toml and following PEP 394

Issue

In the absence of an activated virtual environment (I insist), the poetry add, poetry build, poetry install, poetry lock, poetry remove, poetry run, poetry shell, poetry show and poetry update commands implicitly create a Poetry built-in virtual environment. However currently, the Python interpreter that they include in this virtual environment is the one called by the #!/usr/bin/env python shebang line of the ~/.poetry/bin/poetry script. The problem is that on MacOS and most Linux distributions, /usr/bin/env python resolves to the Python 2.7 interpreter.

So if for instance a user specified in his pyproject.toml file a dependency on the Python 3.7 interpreter:

[tool.poetry.dependencies]
python = "^3.7"

when he runs one of the poetry add, poetry build, poetry install, poetry lock, poetry remove, poetry run, poetry shell, poetry show and poetry update commands for the first time, a RuntimeError is raised:

[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7)
Please activate a compatible Python version.

This is a terrible user experience. Many users have been repeatedly complaining about this since Poetry’s inception, so this issue cannot be ignored: https://github.com/sdispater/poetry/issues/77, https://github.com/sdispater/poetry/issues/305, https://github.com/sdispater/poetry/issues/522, https://github.com/sdispater/poetry/issues/655, https://github.com/sdispater/poetry/issues/721, https://github.com/sdispater/poetry/issues/952, https://github.com/sdispater/poetry/issues/988.

Solution

In https://github.com/sdispater/poetry/issues/988, @Suor made an excellent suggestion that could address this issue:

• add a command that allows the explicit creation of Poetry built-in virtual environments with an option to specify the Python interpreter;
• use the Python interpreter specified in the tool.poetry.dependencies.python property of the pyproject.toml file for the implicit creation of Poetry built-in virtual environments.

In PR https://github.com/sdispater/poetry/pull/731 that will be part of Poetry 1.0.0, @sdispater added a new poetry env command that implements the first point, which is great. Thank you for that @sdispater. But that only addresses half of the problem, the explicit virtual environment creation problem. That does not address the implicit virtual environment creation problem that still raises a RuntimeError.

If we also implemented the second point I think it would greatly improve user experience.

[Edit] Specification for the second point

We should just stick to PEP 394:

Distributors may choose to set the behavior of the python command as follows:

• python2,
• python3,
• not provide python command,
• allow python to be configurable by an end user or a system administrator.

This means that Poetry should only look up python and pythonX in the OS PATH (not pythonX.Y nor pythonX.Y.Z), that is to say Poetry should not be more specific than the major Python version. And since we want to look up the highest Python version for consistency with the dependency lookup mechanism, we should hard code in Poetry the current highest Python major version released (3). With the relatively slow Python release cycle, there will be practically no maintenance: just a number to update every 10 years or so (Python 3 was released in 2008 and Python 4 will not be released before 2020, and possibly never).

So with the following pyproject.toml requirements, Poetry will use the following PATH lookup orders:

• python = ">=3.4.1": [python3, python];
• python = ">=2.6.3": [python3, python2, python];
• python = ">=2.6.3,<3.0.0": [python2, python].

When Python 4 is available, we will update the current highest Python major version released to 4 in Poetry, so the PATH lookup orders of Poetry will become:

• python = ">=3.4.1": [python4, python3, python];
• python = ">=2.6.3": [python4, python3, python2, python];
• python = ">=2.6.3,<3.0.0": [python2, python].

If no pythonX or python is found in the PATH, the following error message will be raised:

[RuntimeError] No current Python version. Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

If a pythonX or python is found in the PATH but its version (given by pythonX -V and python -V respectively) does not match the pyproject.toml requirements, the following error message will be raised:

[RuntimeError] The current Python version (2.7.14) is not supported by the project (^3.7). Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

(Example given with a pyproject.toml requirement of python = "^3.7" and an OS without python3 in the PATH and a python in the PATH pointing to Python 2.7.14.)

[maggyero]

@sdispater

That’s something that I already mentioned before but, for instance, ^3.4 means >=3.4,<4.0 so which Python version should Poetry choose?

Contrary to any other dependencies, the lowest version available I guess. Because otherwise I am not sure how Poetry would do the lookup. For instance with the >=3.4,<4.0 constraint, which file would Poetry look up in the OS PATH environment variable first? python3.9? What if there is a python3.10 or python3.11?

And even if Poetry looked up only by the major version number, it would have the same issue for constraints without upper limits. For instance with the >=3.7 constraint, where would Poetry start? python3? What if there was a python4 or python5? So unless there is a reliable way to get the highest version available, I think we have to choose the lowest version available.

So here are a few examples showing how I see the lookup order (of the lowest version available):

• >=3.4.1: [python3.4.1, python3.4, python3, python];
• >=3.4: [python3.4, python3, python];
• >=3: [python3, python];
• >=2.7.10: [python2.7.10, python2.7, python2, python];
• >=2.7: [python2.7, python2, python];
• >=2: [python2, python];
• >3.4.1: [python3.4.2, python3.4, python3, python];
• >3.4: [python3.5, python3, python];
• >3: [python4, python];
• >2.7.10: [python2.7.11, python2.7, python2, python];
• >2.7: [python2.8, python2, python];
• >2: [python3, python];
• <=3.4.1: [python];
• <=3.4: [python];
• <=3: [python];
• <=2.7.10: [python];
• <=2.7: [python];
• <=2: [python];
• <3.4.1: [python];
• <3.4: [python];
• <3: [python];
• <2.7.10: [python];
• <2.7: [python];
• <2: [python];
• >=3.4.1,<4.0.0: [python3.4.1, python3.4, python3, python];
• >=3.4,<4.0: [python3.4, python3, python];
• >=3,<4: [python3, python];
• >=2.7.10,<3.0.0: [python2.7.10, python2.7, python2, python];
• >=2.7,<3.0: [python2.7, python2, python];
• >=2,<3: [python2, python];
• >3.4.1,<4.0.0: [python3.4.2, python3.4, python3, python];
• >3.4,<4.0: [python3.5, python3, python];
• >3,<4: [];
• >2.7.10,<3.0.0: [python2.7.11, python2.7, python2, python];
• >2.7,<3.0: [python2.8, python2, python];
• >2,<3.0: [].

As you can see, prior to trying python like currently, Poetry will try more specific commands based on the version constraints given in the tool.poetry.dependencies.python property of the pyproject.toml file. What do you think?

Now, maybe we could improve the error message by mentioning the env command.

+1 Yes, definitely.

But before failing with this error message, I think that Poetry should try several commands instead of merely python, using the implicit lookup described above.

[Edit] Forget all of this and see my initial post for the proposed solution.

[David-OConnor]

I understood your original issue as wanting to specify the environment’s Python version version using pyproject.toml.

The current Python version (2.7.14)

current Python version is a trap, when you can (and commonly do on Linux) have multiple versions of Python installed. I’m in favor of specifying a version or semvar requirement in pyproject.toml, using that version if available, and displaying an error if not.