Add documentation for setting environment variables (e.g. for installing extra packages)

I could be wrong, but this seems like a fairly important detail in some deployments. Creating custom OS images with Packer is obviously better but that’s a fairly high bar for a lot of users, probably. I found that this is currently supported with the following caveats:

1. It cannot be configured in cloudprovider.yaml or environment variables on the client (i.e. DASK_CLOUDPROVIDER__GCP__ENV_VARS would be ignored)
2. If passed directly in code, it must be quoted. For example, GCPCluster(..., env_vars=dict(EXTRA_CONDA_PACKAGES="numba xarray")) will not work because the render code (here) does not quote values and since the docker run statement built is already being yaml-escaped via single quotes, the values must be double quoted. The variables must be passed like this: EXTRA_CONDA_PACKAGES="\"numba xarray\"", not EXTRA_CONDA_PACKAGES="'numba xarray'".

Documentation for new users on this would be very helpful. Allowing the env_vars to come from cloudprovider.yaml would also be nice.

[jacobtomlinson]

Have you used mamba much yet?

A little, it seems nice. I think the performance increases here will help things, but my worry is about reproducibility. The times I’ve tried to dump out a conda environment and recreate it recently have failed because packages go missing from anaconda.org. Especially when developing on nighly builds.

Thing conversation reminds me of this meme.

[eric-czech]

Things like Python, Conda, Git, Bash, etc. All of these are necessary tools in order to do your craft. I personally feel like Docker has joined that list in the last few years and learning it is part of the cost of doing business. However, I also feel like as a community we should be making this list smaller, not larger.

Nicely put. Smaller would be great but maybe the need for tools with layered complexity (e.g. numpy -> dask -> xarray) is inevitable to meet the maximum sophistication of a larger number of users. I always imagine that scientists collect at the ends of those spectrums though – I wonder how true that is.

Projects like repo2docker exist to try and create a simple bridge from conda to docker. Perhaps we should also create something like repo2machine which continues to the level that packer does.

Oh interesting, had not seen that before. I like the caching metaphor. It still surprises me a bit that installation of compiled packages via conda can’t be made to be faster than the download and execution of an entire containerized OS image that contains all those same packages. Have you used mamba much yet? I have not at all, but I’m curious if you see that as likely to close the gap.