wandb loads a very outdated `prompt_toolkit` - breaks other ipython modules

Currently wandb supplies a very outdated prompt_toolkit ver 1.0.15 (current is 3.0.8) and messes up with sys.path in util.vendor_setup by adding wandb/vendor/ to it. And pushing it first 😦

This results in:

python -c "import wandb, ipyexperiments"
/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor
/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0
/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor/graphql-core-1.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/ipyexperiments/__init__.py", line 1, in <module>
    from .ipyexperiments import *
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/ipyexperiments/ipyexperiments.py", line 3, in <module>
    import ipykernel # not using it directly, but it needs to be loaded early to avoid crushes in non-jupyter env
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/ipykernel/__init__.py", line 2, in <module>
    from .connect import *
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/ipykernel/connect.py", line 13, in <module>
    from IPython.core.profiledir import ProfileDir
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/IPython/__init__.py", line 56, in <module>
    from .terminal.embed import embed
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/IPython/terminal/embed.py", line 16, in <module>
    from IPython.terminal.interactiveshell import TerminalInteractiveShell
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 21, in <module>
    from prompt_toolkit.formatted_text import PygmentsTokens
ModuleNotFoundError: No module named 'prompt_toolkit.formatted_text'

I recommend:

1. please remove 'prompt_toolkit` from your source code - there is dependencies for that. there is no way you will be remembering to keep it up-to-date and the user may have a need to use a different version anyway - so please don’t take control away from the user.
2. if you have to mess with sys.path, please consider adding those additional paths at the end of it, and not via:

            sys.path.insert(1, p)

this is probably the easiest quick fix - but perhaps it’d help to edit sys.path just for the duration of loading these specific packages and then restoring it back to normal?

currently you push in front of it:

/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor
/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0
/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor/graphql-core-1.1

again, if you need those please consider using dependencies. I see you’re using some really old versions of those libraries, so this is just a recipe for a conflict.

If you have to use those specific versions please change your API for a user to explicitly load any of these vendor libraries and do not load those automatically on import wandb - or change sys.path - if some module chooses to import wandb it shouldn’t negatively impact the rest of the system. I hope this is a reasonable request.

why do I have this issue in first place? Since I tried to use an ipython software along with transformers, and since the latter loads wandb, well:

python -c "import transformers, ipyexperiments"

fails identically as import wandb, ipyexperiments

Currently I use on of the following workarounds:

1. remove wandb
2. load ipykernel first python -c "import ipykernel, wandb, ipyexperiments" which gets the right prompt_toolkit
3. load wandb or transformers last: python -c "import ipykernel, wandb, ipyexperiments" - same effect as (2)
4. funnily enough installing comet_ml solved this too since transformers loads it before wandb and comet_ml loads ipykernel first - same effect as (2)

Thank you!

[hueds]

I also ran into this error when I try to use ipdb package for debugging.

[vanpelt]

@dreamflasher I *believe we fixed this in the most recent release 0.10.11, but I haven’t personally confirmed this. Can you give it a try and provide a snippet if it’s still not working for you?