(feature request) More support for conditional installation

First of all, apologies if what I’m requesting is already possible! I did my best to search for anything relevant.

Context

We make use of Tensorflow in our codebase, which has the annoying quirk that there is a GPU-enabled wheel called “tensorflow-gpu” and a CPU-only wheel called “tensorflow”. The GPU-enabled package will crash on a system without the CUDA libs installed. The wheels also share most of their package structure and seem to clobber each other if you accidentally install both. Some of that behaviour is probably a bit un-Pythonic, but that’s what has been shipped.

Now, ideally Google would build a single wheel that gracefully falls back to CPU-only if CUDA insn’t installed, but that probably isn’t happening any time soon. Their official solution seems to be not to specify TF as a dependency for anything, and require the user to manually install the correct build [1]. This is less-than-ideal for configuration management.

Feature Request

I’d very much like it if I could set up one Pipfile that defaults to installing the CPU package unless I set some custom flag indicating GPU is available.

Discussion

The closest existing functionality I’m aware of is the support for environment markers - if there was an extension to allow user-defined markers then this could be a fairly clean solution. I’m unsure how pipenv handles the lockfiles when there are multiple possible configurations, but this would continue working the same way.

As a rough example of what I’m suggesting: Pipfile:

[packages]

tensorflow-gpu = { version = "*", gpu_enabled = "=='1'" }
tensorflow = { version = "*", gpu_enabled = "==''"  }

Installing CPU version: pipenv install Installing GPU version: pipenv --env_marker "gpu_enabled=1" install

[1] https://github.com/deepmind/sonnet “Sonnet will work with both the CPU and GPU version of tensorflow, but to allow for that it does not list Tensorflow as a requirement, so you need to install Tensorflow separately if you haven’t already done so.”

EDIT: Another downside to Google’s approach that just occurred to me: manually installing tensorflow will really not work well with pipenv - every time the env gets purged, tensorflow will need to be manually installed again.

[ed-alertedh]

@Froskekongen I don’t really want to go down that route. “gpu_enabled” seems like a fairly ill-defined ad-hoc flag to add to a multi-platform standard. “cuda_enabled” would be easier to define but is even more ad-hoc - I suspect the suggestion would be thrown out even if we did try to get it into the spec.

Also it turns out “extras” would not help here - that just allows a package to specify optional dependencies.

Basically, I can already tell pipenv to only install a package if it’s running on Windows. The best example of this in our codebase is:

numpy = { version = "==1.13.3", os_name = "!='nt'", index="pypi" }
"c500f7b" = {path = "numpy-1.13.3+mkl-cp36-cp36m-win_amd64.whl", os_name = "=='nt'"}

My original request was simply a way to do the same thing, except instead of looking at os_name, it would look at a user-defined variable.

[Overdrivr]

I realize this could be also useful between dev and test environment for modules.

In dev mode, the module must be installed in editable mode -e . to make development easy. However in test environment (CI test runner like Jenkins or GitlabCI), it would be better to install to install the package directly from the wheel, to be closer to how the package will be used in reality.

AFAIK, it is not possible in a Pipfile to specify whether to install a package from editable or wheel. The only solution I see is to manage 2 Pipfiles, which is not convenient.

@kennethreitz Any recommendations for this kind of scenario ?