Support for build variants/profiles

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

For some Python packages, there are different wheel variants with identical package names, which are distinguished only based on PEP-440 local version labels (the label in 1.2.3+label). E.g. for PyTorch or mxnet, this is used to deliver different variants that use CPU or one of several CUDA versions. This is independent of system or hardware platform and Python version - PyTorch have a wheel for almost every combination of these. For example, for PyTorch 1.5.1 with Python 3.8 on Linux there are versions pytorch-1.5.1 (for CUDA 10.2), pytorch-1.5.1+cu101 (for CUDA 10.1), pytorch-1.5.1+cu92 (for CUDA 9.2) and pytorch-1.5.1+cpu (for CPU).

As far as I know, the only way Poetry supports for switching between these variants on a given platform is to modify the pyproject.toml file, which obviously is not so optimal. Markers are not a solution here because it is not possible switch between different variants on the same machine. Neither are optional dependencies and extras, because the package names are identical for the different variants.

In the Java world, Maven offers build profiles, which allow for specifying different [sets of dependencies, build plugins etc… Maven profiles can be activated and deactivated on a per-build basis (or using triggers similar to markers, but that is not interesting for this use case) and independently of each other. Maven profiles are a bit complex and bloated, but I think picking and modifying some of its ideas for this specific use case could be useful for Poetry.

I’d suggest the following syntax in pyproject.toml:

[tool.poetry]
name = "myproject"
version = "1.2.3"
...

[tool.poetry.dependencies]
# Global dependencies (for all profiles)
...


[[tool.poetry.profile]]
name = "cuda-10.1-backend"
# Suggested syntax for specifying different local versions for different profiles
localversion = "cuda101"

[tool.poetry.profile.dependencies]
# Specific dependencies for CUDA 10.1 profile
torch = [
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]


[[tool.poetry.profile]]
name = "cpu-backend"
localversion = "cpu"

[tool.poetry.profile.dependencies]
# Specific dependencies for CPU profile
torch = [
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]

There could be one lock file per profile/combination of profiles (which I think would be easier to understand and maintain, and probably also to implement) or one global lockfile which combines several profiles/profile combinations. The profiles could be activated using a CLI switch --profile/-p for poetry update/lock/install/add/etc.; E.g. poetry lock -p cpu-backend would only lock the versions for the CPU variant, and poetry install -p cpu-backend would install the global packages and those defined in the cpu-backend lockfile.

Another variant would be profile groups where one of several alternatives must be active, and one of them may be active by default:

[tool.poetry]
name = "myproject"
version = "1.2.3"
...

[tool.poetry.dependencies]
# Global dependencies (for all profiles)
...


[[tool.poetry.profilegroup]]
name = "backend"

# Either cuda-10.1 or cpu must be activated during lock/install/update, cpu is active by default, but remains deactivated if cuda-10.1 is activated
[[tool.poetry.profilegroup.profile]]
name = "cuda-10.1"
localversion = "cuda101"

[tool.poetry.profilegroup.profile.dependencies]
# Specific dependencies for CUDA 10.1 profile
torch = [
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cu101/torch-1.5.1%2Bcu101-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]


[[tool.poetry.profilegroup.profile]]
name = "cpu"
default = true
localversion = "cpu"

[tool.poetry.profilegroup.profile.dependencies]
# Specific dependencies for CPU profile
torch = [
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-win_amd64.whl", platform="win32"},
 {url="https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp37-cp37m-linux_x86_64.whl", platform="linux2"}
]

For the profile group variant the syntax could be “-p backend=cpu” etc.

An alternative solutions might be allowing to specify constraints under extras, but I don’t think that’s the point of extras.

[tarasivashchuk]

Anything on this? I always have to create a separate script to install PyTorch because of this.

[vedosis]

At least within pyproject.toml I can get just torch to install via:

[[tool.poetry.dependencies.torch]]
python = "^3.8"
markers = "platform_system == 'Linux'"
url = "https://download.pytorch.org/whl/lts/1.8/cpu/torch-1.8.1%2Bcpu-cp38-cp38-linux_x86_64.whl"

However, the version comes across as 1.8.1+cpu. So trying to install torchvision in the same way fails because:

  SolverProblemError

  Because torchvision (0.9.1+cpu) depends on torch (1.8.1)
   and librephotos depends on torch (1.8.1+cpu), torchvision is forbidden.
  So, because librephotos depends on torchvision (0.9.1+cpu), version solving failed.

Any concerns with version to be specified and for that value to override what’s discovered for dependency resolution?

Alternatively, dropping +label during version resolution would work for me.

I’m happy to do the implementation if you don’t have spare cycles to do it.