Source environments from external files

I have some environment definitions which are quite large with many packages. It would be convenient to have the option to source any such environment from a separate file, rather than including the environment inline within qhub-config.yaml.

How about if the environments section in the config file allowed null values, which means that the content should be pulled from a file with that name in the repository root directory?

environments:
  "environment-special.yaml":
    # null value allowed, meaning load from separate file

It looks like it should be doable with some modifications to the source code around here:

https://github.com/Quansight/qhub/blob/4cc393ec8ba50fd850b0f07781446229138809f0/qhub/template/hooks/post_gen_project.py#L24

[brl0]

I also like and prefer the idea of externalized environment files.

for more complicated environments, although I can solve and install them locally, they fail to solve and install on the deployed qhub service, even after waiting many hours

I have encountered this behavior as well and have adopted the same approach as a work around.

An idea just occurred to me, is that maybe a good workflow for qhub/conda-store would be to solve the changed environments in the render phase or in CI prior to deployment, exporting the pinned environments to a managed folder. This would help catch failed environments earlier and more explicitly, rather than the current experience of deploying and then waiting and hoping the environment solves. @costrouc, I’d be eager for your thoughts on this idea.

[alimanfoo]

Thinking about this some more, FWIW I think it could be better to just let users manage environments by controlling the files in the environments directory themselves. I.e., remove the environments section from qhub-config.yaml, and during deployment figure out what environments are present by inspecting the environments directory…