Question about conda-meta exclusion

I notice that I can create a non-base conda environment that includes conda>4.5, pack this environment, and later unpack it without issue.

conda create -n pack-base python=3 conda pip

# normal conda output removed for brevity ...

conda pack -n pack-base --arcroot pack-base
Collecting packages...
Packing environment at '/Users/parente/miniconda3/envs/pack-base' to 'pack-base.tar.gz'
[########################################] | 100% Completed |  5.0s

# on another host
tar xzf pack-base.tar.gz -C tmp
source /tmp/pack-base/bin/activate
conda unpack

I notice that conda-meta is not included by default in the packed environment, which means conda list (and maybe other conda commands?) does not function after unpacking.

conda list

EnvironmentLocationNotFound: Not a conda environment: /tmp/pack-base

However, if I manually add the conda-meta from the original environment to the tarball so that it’s available later, conda seems to work properly after unpacking. So much so that I can list, install, uninstall, etc. packages in that unpacked environment.

Could you explain why conda-meta is not included in the tarball generated by conda pack? I’m guessing there’s a good reason, and that if I add conda and conda-meta to my packed envs, I’ll hit some surprise down the line that I’ve not discovered yet.

I ask because I have an interest in packing and distributing environments in which conda itself functions.

[jcrist]

Whether intentional or not, with conda 4.5.x, I can install conda in a non-root/base environment.

Ah, good to know (I should really read the conda release notes). In this case, I think the todo list for this functionality is:

• Stop removing conda and conda-meta from the environment (looks like new conda doesn’t put a conda file in /bin unless conda was explicitly installed in the environment.
• Rewrite the json files in conda-meta to fix the absolute path and link information
• Tests (if you can’t figure out an easy way to test this functionality, I’d be happy to handle this).

Thanks for looking in to this!

[jcrist]

Apologies for the delay here, still catching up after vacation.

The conda-meta directory (and a few other conda-specific things) were stripped from the environment because the intent of this project was to distribute conda environments to locations where conda wasn’t already installed (my use case specifically was deployment on YARN). Since this doesn’t handle the root environment, and conda can only be installed in the root environment, this can’t be used to distribute environments that contain conda. And if a machine already has conda installed on it, it seems cleaner and more robust to try and recreate the environment using something like conda env.

That said, I don’t see any harm in not stripping out conda-meta. The file sizes are small and should compress well, and users can always filter it out if needed. A few potential issues that come to mind (untested):

• We also strip out conda, activate, and deactivate from /bin, and replace activate and deactivate with our own versions. This may break workflows using standard conda tools, or it may not. Unsure.

• The conda-meta files include a couple absolute paths (mostly package cache information). I think these are harmless to copy over (conda is fairly robust to malformed package caches), but the best thing to do would probably be to fix link.type in all files to be copy (same for pathtype for each path record), and replace the absolute paths with empty strings or some bogus string (to strip the user information).

I don’t have time currently to test these issues out, but if you do @parente I’d accept and review a PR.