Multi Platform support for Pipfile.lock

Is your feature request related to a problem? Please describe.

Currently it is not possible to create a Pipfile.lock for multiple platforms automatically. Only hashes for the current platform are locked. This is problematic when you want to use Pipefile.lock to share locked dependency versions in a multi platform setup, e.g. development on Windows or Mac and production on Linux.

E.g. cryptography generates the following Pipfile.lock entry on Windows 10 64bit:

        "cryptography": {
            "hashes": [
                "sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==37.0.2"
        },

and the following one on a Linux VM:

        "cryptography": {
            "hashes": [
                "sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
                "sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
                "sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==37.0.2"
        },

Describe the solution you’d like

My preferred solution would be a new entry in the Pipfile that defines which platforms hashes must always be included when locking. If the current systems platform isn’t in the list it should be be added to the lock list though to retain backwards compatibility.

The Pipfile entry could look like this:

[[source]]
platforms = "manylinux_2_24_x86_64 or win_amd_64"

If possible it would be great if the values could be eagerly validated and an error message printed on invalid values.

Note: Ticket https://github.com/pypa/pipenv/issues/210 is similar to this one, but I was asked to open a new ticket

[thehesiod]

I think it’s worth looking at the poetry file format for this type of scenario:

[package.dependencies]
numpy = [
    {version = ">=1.18.5", markers = "platform_machine != \"aarch64\" and platform_machine != \"arm64\" and python_version < \"3.10\""},
    {version = ">=1.19.2", markers = "platform_machine == \"aarch64\" and python_version < \"3.10\""},
    {version = ">=1.20.0", markers = "platform_machine == \"arm64\" and python_version < \"3.10\""},
]

along with

numpy = [
    {file = "numpy-1.23.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:aeba539285dcf0a1ba755945865ec61240ede5432df41d6e29fab305f4384db2"},
    {file = "numpy-1.23.1-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:7e8229f3687cdadba2c4faef39204feb51ef7c1a9b669247d49a24f3e2e1617c"},
    {file = "numpy-1.23.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1408c3527a74a0209c781ac82bde2182b0f0bf54dea6e6a363fe0cc4488a7ce7"},
    {file = "numpy-1.23.1.tar.gz", hash = "sha256:d748ef349bfef2e1194b59da37ed5a29c19ea8d7e6342019921ba2ba4fd8b624"},
]

that way multiple versions are supported, with each platform getting what it requires

[januszm]

I think this is a really important problem to solve right now. For example, if we have a project that uses numpy as a dependency and we have an x86_64 server, Github Actions CI also on x86_64, 1 developer on Linux x86_64, one on older x86_64 Mac and one more with an M1 Macbook arm64. Each time someone updates the lockfile, at least one person has problem. At the moment, one quick solution I found was to manually modify the Pipfile.lock file and add this (for numpy):

"markers": "(platform_machine == 'x86_64' or platform_machine == 'arm64') and python_version < '3.10'",

in

"numpy": {
  "hashes": [...],
  "markers": "...",
  "version": "==1.23.2"

(the above allows me to work on M1 Mac and run CI on x86_64 Github Action + deploy to amd64 Linux server, but it’s not sustainable as next time someone adds or updates a package it’ll be lost)

The Ruby Bundler solves this problem by adding separate lock file entries for each platform, e.g.

nokogiri (1.13.6)
nokogiri (1.13.6-aarch64-linux)
nokogiri (1.13.6-x86_64-darwin)
nokogiri (1.13.6-x86_64-linux)