Multi-architecture wheel selection across disperate sources (with markers)

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04 on both aarch64 and x86_64

• Poetry version: 1.1.12

• Link of a Gist with the contents of your pyproject.toml file: (see below for variants)

Issue

I’m trying to build a project that will be used in multi-architecture environments (aarch64 and x86_64 specifically). One of my project’s dependencies (jaxlib) only offers prebuilt wheels for x86_64 via PiPy. As a solution, I’ve built and hosted the wheel for aarch64 separately, which necessitates adding secondary dependency sources in pyproject.toml like so:

[tool.poetry]
name = "temp"
version = "0.1.0"
description = ""
authors = ["merberich <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.9"
cupy-cuda102 = "^10.1.0"
jaxlib = [
  { version = "0.1.71" },
  { url = "<valid-url-to-aarch64-wheel-host>" }
]
jax = "0.2.20"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Building a new poetry.lock for this project produces jaxlib references:

cat poetry.lock | grep jaxlib
cpu = ["jaxlib (==0.1.71)"]
cuda102 = ["jaxlib (==0.1.71+cuda102)"]
cuda111 = ["jaxlib (==0.1.71+cuda111)"]
minimum-jaxlib = ["jaxlib (==0.1.69)"]
tpu = ["jaxlib (==0.1.71)", "libtpu-nightly (==0.1.dev20210809)", "requests"]
name = "jaxlib"
jaxlib = [
    {file = "jaxlib-0.1.71-cp37-none-macosx_10_9_x86_64.whl", hash = "sha256:ad36895ceb68782bb74f0657da127cece06c19ce1e72ad9c660f79bce549618e"},
    {file = "jaxlib-0.1.71-cp37-none-manylinux2010_x86_64.whl", hash = "sha256:059eb572121e3f13e3b841c07137a72f3d0aeb76dc6ddf178922a327994f60b8"},
    {file = "jaxlib-0.1.71-cp38-none-macosx_10_9_x86_64.whl", hash = "sha256:c823b65c95aa7b6d8eb4a45bc6389d43c8d6dd20fe0c531345753819c70cff54"},
    {file = "jaxlib-0.1.71-cp38-none-manylinux2010_x86_64.whl", hash = "sha256:8f4447d6053b55bab9565e365b60c80ff186b0cb05407146ea844d328eaba2bf"},
    {file = "jaxlib-0.1.71-cp39-none-macosx_10_9_x86_64.whl", hash = "sha256:f5fc1873c25a7b07f9406bcb09540cef6e2b151aa139be52e34cf35f7b8390b0"},
    {file = "jaxlib-0.1.71-cp39-none-manylinux2010_x86_64.whl", hash = "sha256:c66a1bdb57934093938fd6e1767216c198bd1a102b8f5636fdb1f9a5b0d11067"},

Note that for each of these wheels, the supported architecture is always x86_64… which my aarch64 machine does not use (IMO these wheels should not be considered valid sources on my aarch64 machine). Instead, poetry install loads the secondary source specified in pyproject.toml, which breaks the lockfile boundary:

poetry install
Installing dependencies from lock file

Package operations: 10 installs, 0 updates, 0 removals

  • Installing numpy (1.22.2)
  • Installing six (1.16.0)
  • Installing absl-py (1.0.0)
  • Installing fastrlock (0.8)
  • Installing flatbuffers (2.0)
  • Installing opt-einsum (3.3.0)
  • Installing scipy (1.6.1)
  • Installing cupy-cuda102 (10.1.0)
  • Installing jax (0.2.20)
  • Installing jaxlib (0.1.71 <valid-url-to-aarch64-wheel-host>)

So the real issue here is that the lockfile does not contain the secondary source, which is necessary to build on this machine - instead, poetry install has to reference the pyproject.toml to identify secondary sources, which defeats the purpose of having a lockfile…

Possibly worth note: I’ve also tried to use markers to have the lockfile generation see that both sources are needed to cover all relevant architectures:

jaxlib = [
  { version = "0.1.71", markers = "platform_machine == 'x86_64'" },
  { url = "<valid-url-to-aarch64-wheel-host>", markers = "platform_machine == 'aarch64'" }
]

Unfortunately, this didnt cause any change in results.

Our workaround will be to host an alternative package registry with dependencies provided for both x86_64 and aarch64 (and this way ALL wheels should appear a ‘valid’ sources in the lockfile), but that seems like a lot of extra infrastructure just to account for something the lockfile could/should contain.

I think the correct solution would involve having the lockfile list alternatives for cases where markers are used to specify multiarch environments. The behavior to avoid would be having poetry look outside the lockfile when one exists.

[sneakers-the-rat]

to add to examples:

markupsafe = [
    {version="^2.1.0", markers="platform_machine!='x86_64'", source='piwheels'},
    {version="^2.1.0", markers="platform_machine=='x86_64'", source="pypi"}
]

results in

   0: Duplicate dependencies for markupsafe
   0: Merging requirements for markupsafe (>=2.1.0,<3.0.0)

even though they are definitely not duplicates.

edit: on 1.1.13

[radoering]

The first example (version and url dependency) should already work with poetry 1.2.1. The second example (two version dependencies from different sources) should be fixed by #6679.