Poetry v1.1.6 does not respect markers or version specifiers when generating lockfile

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10 v20H2 build 19042.867
• Poetry version: 1.1.6
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/tigerhawkvok/b7e1e54872983148d6599b1180cac67f

Issue

This worked on my prior Poetry build (possibly as old as v0.12, but I updated today to v1.1.6). Neither the python nor the markers key seems to be respected in pyproject.toml.

An illustrative screenshot:

Full sample: poetry_dep_bug.zip

If I manually edit my lockfile, I can recover the expected behaviour

[[package]]
name = "pycurl"
version = "7.43.0.4"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.6, < 3.7"

[package.source]
type = "file"
url = "pycurl-7.43.0.4-cp36-cp36m-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.4"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.7, < 3.8"

[package.source]
type = "file"
url = "pycurl-7.43.0.4-cp37-cp37m-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.4"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.8, < 3.9"

[package.source]
type = "file"
url = "pycurl-7.43.0.4-cp38-cp38-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.5"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">= 3.9, < 3.10"

[package.source]
type = "file"
url = "pycurl-7.43.0.5-cp39-cp39-win_amd64.whl"

[[package]]
name = "pycurl"
version = "7.43.0.6"
description = "PycURL -- A Python Interface To The cURL library"
category = "main"
optional = false
python-versions = ">=3.5"
marker = "platform_system != 'Windows'"

[tgolsson]

I’ve got a fork that’s solving some of the issue re: missing markers; https://github.com/python-poetry/poetry/compare/master...tgolsson:ts/markers-in-lockfile. However, while doing the final cleanup and fixing the last tests I noticed that this was part of an intentional change by @sdispater two years ago, #2361.

While I understand the motivations of that PR and (some of) what it solves, it leads to forgetting about “top-level” dependencies with markers in the actual pyproject.toml. My PR adds those back and then makes the version solver deal with them again (likely in a far too simplistic manner in light of the original PR). However, the lockfiles that are currently generated without my changes are broken if I’m using markers to toggle versions in my pyproject.toml:

torch = [
    { version = "1.10.2+cu111", markers = "sys_platform != 'darwin'" },
    { version = "1.10.2+cpu", markers = "sys_platform == 'darwin'" }
]

Without the proper information in the lockfile both torch deps are identical.

One alternative solution route would be to generate the project itself into the lockfile, so if our project is foo we’d depend on both, with the proper markers at that level. However, that is a bigger change, and I’m quite sure it’d be both a breaking change in compatibility between poetry.lock and poetry, in both directions.

I realize a lot of time has passed @sdispater, but do you remember enough to have any input on solutions here?

(Edit: I think that the issue @rschmied posted above is different from markers for primary dependencies, and my fix would have no effect. That seems like a bug in the constraint solving itself.)

[tgolsson]

OK; so more investigation and… On latest poetry it seems like if I ensure that there’s no overlap anywhere in my versions; it works out fine. I was getting tricked by my fixes because I’d started with one case and then started changing Poetry to fix what was likely an error in my pyproject.toml, and then I eventually fixed that along the way but kept digging.

So - I started with the following:

torch = [
    { version = "1.10.2+cu111", source="foo", markers = "sys_platform != 'darwin'" },
    { version = "1.10.2", markers = "sys_platform == 'darwin'" }
]

And this indeed exhibits some bad behavior, at least when toggling which version belongs on which platform. In particular, 1.10.2+cu111 is a valid subversion for 1.10.2, so there’s no fundamental difference between a naked 1.10.2 and the cu111. So, that’s what I tried to solve. Later on after doing more experiments I’d unwittingly switched to the +cpu version for some other testing to see if using another source would work better. This changed the version resolution functionality, because the local tags are different.

I thus think this issue is aimed at solving the wrong problem – the markers not being in poetry.lock isn’t the source of this problem because they are seemingly read from pyproject.toml. I think there’s good reasons to have them in poetry.lock for other reasons, but my testing over the last two days conclude that it’s not required for poetry install to correctly work with markers.

Now, one important difference between this and OP’s post is that I’m using 1.2.0b1 to test, and the issue was reported for 1.1.6. However, there’s clearly bugs with version resolution still – for example, if I test

torch = [
    { version = "1.10.2+cu111", source="foo", markers = "sys_platform != 'linux'" },
    { version = "1.10.2", source="bar", markers = "sys_platform == 'linux'" }
]

It’ll helpfully offer only +rocm local tags… Not quite what I had in mind. This might be related to @rschmied’s issues above.

Anyways, long story short: I went into a rabbit hole hunting a one year old bug, did a lot of work that was unnecessary, and finally undid everything. As redemption I tested your zipfile @tigerhawkvok and with fresh installed poetry==1.2.0b1 and Python 3.8, I get

$ poetry install
Creating virtualenv poetry-dep-bug-XteSQS2G-py3.8 in <snip>\AppData\Local\pypoetry\Cache\virtualenvs
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing pycurl (7.43.0.4 <snip>/poetry_dep_bug/pycurl-7.43.0.4-cp38-cp38-win_amd64.whl)

Which seems to be what I should expect.