Replace `freeze.py` with `conda-lock`

Proposed change

We use freeze.py to generate fully specified environment.yml files out of our environment.yml files. It creates a docker image, installs everything, and then lists out the installed packages.

Instead, we could use conda-lock. This resolves environment.yml files to fully specified lock files we can use. It doesn’t install the packages either, so should be much faster.

Alternative options

Do nothing! Our current solution works well.

Who would use this feature?

Developers of repo2docker who modify environment.yml files. Currently the freezing process can take a while - this would make it faster.

How much effort will adding it take?

We need to investigate if conda-lock will actually do what we want. After that, implementing it shouldn’t be too difficult - a couple hours of work for someone very familiar with the codebase, maybe 8-12h for someone who isn’t.

Who can do this work?

1. Decent python skills
2. A vague understanding of how conda’s environment.yml files work

[A-Fisk]

Hello. First time contributer, saw you mention this on twitter so thought I’d have a look. Feel like I’ve got no idea what I’m doing so help appreciated.

I’ve had a look around freeze.py and conda lock and I think I understand what they both do. This is my current understanding:

• freeze.py - takes environment.yml file and generates environment.*.frozen.yml files
• app.py - then at main runtime when it uses CondaBuildPack class it selects the right environment.*.frozen.yml file and runs this as part of install-miniforge.bash
• conda-lock generates conda-system.lock file which can then be used to create an environment using conda create -f conda-system.lock
• to switch to using conda-lock we need to replace where conda env create -f environment.yml is run with conda create -f conda-system.lock
• from having a bit of a dig this seems to be done in install-miniforge.bash line 42-44:

echo "installing notebook env:"
cat /tmp/environment.yml
conda env create -p ${NB_PYTHON_PREFIX} -f /tmp/environment.yml

• I’m stuck trying to figure out the system for selecting the right environment file and how that is then passed as an argument into install-miniforge.bash and then ends up as /tmp/environment.yml
• I think it’s something in the get_build_scripts function in CondaBuildPack but I’m not sure.

Extra question:

1. Workflow for using freeze.py? I’m trying to figure out where this fits in. If I’ve got it right it’s run whenever the base environment.yml file is updated and then that updates all the frozen files.

Anyway, that’s all I can come up with for now. I hope I’m on the right track and haven’t got it all completely wrong. Not sure how useful this is. Not sure if I’ve described things in too much detail or not enough, but that’s what I’ve found for now. Hope to come back to this tomorrow.

[xhochy]

Being more explicit: the explicit URLs

😆 Thanks! Yeah, the “explict” URL is a direct path to the artifact, no matter where it is.

Actually: no 😛 It still includes the label but nowadays, we don’t remove the main label anymore from a package, we just patch it out of the repodata. Thus this trick really only works with the conda-forge channel.