Setting up conda env takes 2-8min per CI job — cache it?

Looking at a recent CI run on main, I see the Setup Conda Environment step taking 2-3min on linux, and typically 6-7min on Windows and macOS. That’s 10-15% of total job runtime.

Could we cache the conda environment between CI runs, and only reinstall when the environment.yaml changes? (And maybe once per day too, so things don’t get too stale?)

A downside is that most of our CI dependencies are un-pinned, which we’re relying on as a crude form of upstream testing: when dependencies release new versions, our CI might non-deterministically go red if they broke something. Personally, I think this is a bad experience for contributors and bad practice for CI in general, but it doesn’t seem like we’re going to change this right now https://github.com/dask/community/issues/203. So if we did cache conda envs, it might add more latency when upstream dependencies break us.

cc @fjetter @quasiben

[charlesbluca]

Yeah I can look into this

[charlesbluca]

Could we have a nightly job that installs the env, runs the tests and only caches it when everything passes?

Think this would already be happening if we merge in #6855, as testing CI is also running on a schedule, and thus a cached environment would be created (if it doesn’t already exist) at UTC 6:00 and 18:00:

https://github.com/dask/distributed/blob/f43bc4729f52b3fbfc0758506adfcf51276ffb60/.github/workflows/tests.yaml#L7

I’m not sure if these times were set with Dask maintainers’ local timezones in mind, but it would be nice to switch the scheduled CI to run at UTC 00:00 and 12:00, so that new cached environments start getting built right when the old cached envs become stale.