Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Question about conda-meta exclusion

See original GitHub issue

I notice that I can create a non-base conda environment that includes conda>4.5, pack this environment, and later unpack it without issue.

conda create -n pack-base python=3 conda pip

# normal conda output removed for brevity ...

conda pack -n pack-base --arcroot pack-base
Collecting packages...
Packing environment at '/Users/parente/miniconda3/envs/pack-base' to 'pack-base.tar.gz'
[########################################] | 100% Completed |  5.0s

# on another host
tar xzf pack-base.tar.gz -C tmp
source /tmp/pack-base/bin/activate
conda unpack

I notice that conda-meta is not included by default in the packed environment, which means conda list (and maybe other conda commands?) does not function after unpacking.

conda list

EnvironmentLocationNotFound: Not a conda environment: /tmp/pack-base

However, if I manually add the conda-meta from the original environment to the tarball so that it’s available later, conda seems to work properly after unpacking. So much so that I can list, install, uninstall, etc. packages in that unpacked environment.

Could you explain why conda-meta is not included in the tarball generated by conda pack? I’m guessing there’s a good reason, and that if I add conda and conda-meta to my packed envs, I’ll hit some surprise down the line that I’ve not discovered yet.

I ask because I have an interest in packing and distributing environments in which conda itself functions.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:8

github_iconTop GitHub Comments

jcristcommented, Aug 3, 2018

Whether intentional or not, with conda 4.5.x, I can install conda in a non-root/base environment.

Ah, good to know (I should really read the conda release notes). In this case, I think the todo list for this functionality is:

  • Stop removing conda and conda-meta from the environment (looks like new conda doesn’t put a conda file in /bin unless conda was explicitly installed in the environment.
  • Rewrite the json files in conda-meta to fix the absolute path and link information
  • Tests (if you can’t figure out an easy way to test this functionality, I’d be happy to handle this).

Thanks for looking in to this!

jcristcommented, Aug 3, 2018

Apologies for the delay here, still catching up after vacation.

The conda-meta directory (and a few other conda-specific things) were stripped from the environment because the intent of this project was to distribute conda environments to locations where conda wasn’t already installed (my use case specifically was deployment on YARN). Since this doesn’t handle the root environment, and conda can only be installed in the root environment, this can’t be used to distribute environments that contain conda. And if a machine already has conda installed on it, it seems cleaner and more robust to try and recreate the environment using something like conda env.

That said, I don’t see any harm in not stripping out conda-meta. The file sizes are small and should compress well, and users can always filter it out if needed. A few potential issues that come to mind (untested):

  • We also strip out conda, activate, and deactivate from /bin, and replace activate and deactivate with our own versions. This may break workflows using standard conda tools, or it may not. Unsure.

  • The conda-meta files include a couple absolute paths (mostly package cache information). I think these are harmless to copy over (conda is fairly robust to malformed package caches), but the best thing to do would probably be to fix link.type in all files to be copy (same for pathtype for each path record), and replace the absolute paths with empty strings or some bogus string (to strip the user information).

I don’t have time currently to test these issues out, but if you do @parente I’d accept and review a PR.

Read more comments on GitHub >

github_iconTop Results From Across the Web

conda/conda-build - Gitter
so how does one specify an exclude? the only reference I found to "exclude" ... and source path: pointed to that "one dir...
Read more >
Defining metadata (meta.yaml) - Conda
Used to exclude some or all of the files in the build recipe from the list of ... In that case, the file...
Read more >
Conda incomplete creation of virtual environment after ...
bin conda-meta ... I had a small follow-up question though. ... @Anando You should ask a new question and include all of the...
Read more >
Knowledge Base — conda-forge 2022.12.21 documentation
SS64's CMD howto pages are the best resource for any kind of question regarding ... but not on packages that are excluded from...
Read more >
Why You Need Python Environments and How to Manage ...
Create a file named pinned in the environment's conda-meta directory. ... A little side note based on one of my reader's question (thanks ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found