Ambiguous instructions for installing python2 and python3 kernels, Big Split path issues?

In attempting to collaborate with @suchow, we ran into an issue with his jupyter notebook having been installed through python2 and my having written the notebook with a python3 kernel in mind.

He had an ipython3 version of the notebook with python3 as its kernel, but that would not run Python 3.5.

In trying to figure out how to fix the original problem (i.e., a jupyter notebook installed through python2 not having access to a python3 kernel) I began a long trek through a lot of conflicting documentation, this issue is the trail as I tried to recreate it (I am guessing I left out some parts).

I ended up at this documentation which only says that a Python kernel is installed when you installed the notebook https://github.com/jupyter/jupyter/blob/master/docs/source/install.rst.

Look at http://jupyter.cs.brynmawr.edu/hub/dblank/public/Jupyter Help.ipynb#1.4.2-Enable-Python-3-kernel but that seems to use the ~/.ipython/kernels/ path rather than the post Big Split path which seems to be ~/Library/Jupyter or /usr/local/share/jupyter (since ~/.jupyter is for config files, not data files and kernels are data files).

Eventually I found http://jupyter.readthedocs.org/en/latest/migrating.html#kernels, which says

You can see where the file is with jupyter kernelspec list, and then modify kernels/python3/kernel.json by hand. a2km is an experimental project that tries to make these things easier.

But the file path is not returned when I run jupyter kernelspec list, all it does is return

Available kernels:
  python3

And going from to https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages includes no instructions for how to install a Python 2 kernel if you began with a Python 3 kernel (or vice-versa).

I only noticed now that

IPython.kernel → jupyter_client, ipykernel (IPython.kernel became two packages - one for the client-side APIs, one for the IPython kernel for Jupyter)

which leads me to look at http://ipython.org/ipython-doc/dev/development/kernels.html

Which suggests (contrary to the above jupyter migration documentation)

To see the available kernel specs, run:

   ipython kernelspec list

(which also only returns a list of the kernels with no path)

And also says:

<colgroup> <col width="10%"/> <col width="47%"/> <col width="43%"/> </colgroup>

	Unix	Windows
System	<tt class="docutils literal"><span class="pre">/usr/share/jupyter/kernels</span></tt> <tt class="docutils literal"><span class="pre">/usr/local/share/jupyter/kernels</span></tt>	<tt class="docutils literal"><span class="pre">%PROGRAMDATA%\jupyter\kernels</span></tt>
User	<tt class="docutils literal"><span class="pre">~/.ipython/kernels</span></tt>

which conflicts with the above claim that it should be in either /usr/local/share/jupyter/kernels or ~/Library/Jupyter (if local) on OSX Unix systems, and suggests continued use of the ~/.ipython/kernels path (which I should mention, does not exist on my machine).

I’m genuinely baffled as to how to either instruct someone else to install a python3 kernel on their machine or to install a python2 kernel on mine…

Just to check these are the errors I got when I tried to run

jupyter kernelspec install python2
Traceback (most recent call last):
  File "/usr/local/bin/jupyter-kernelspec", line 8, in <module>
    main()
  File "/usr/local/bin/jupyter-kernelspec", line 5, in main
    KernelSpecApp.launch_instance()
  File "/usr/local/lib/python3.4/site-packages/traitlets/config/application.py", line 592, in launch_instance
    app.start()
  File "/usr/local/lib/python3.4/site-packages/jupyter_client/kernelspecapp.py", line 177, in start
    return self.subapp.start()
  File "/usr/local/lib/python3.4/site-packages/jupyter_client/kernelspecapp.py", line 113, in start
    replace=self.replace,
  File "/usr/local/lib/python3.4/site-packages/jupyter_client/kernelspec.py", line 208, in install_kernel_spec
    shutil.copytree(source_dir, destination)
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shutil.py", line 296, in copytree
    names = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: 'python2'

Which returned similar errors when run with python2.7,python27,PYTHON2,PYTHON2.7,PYTHON27 and even python.

Regardless, the amount of hoops I had to go through and the number of conflicts I ran into suggested to me that this needs more documentation on how to do.

I’m happy to help write the necessary changes in all the places I found (plus a number of stackexchange answers that I think may be inconsistent with the Big Split… I’ve posted links to at the bottom of this Issue), I just need to know how to solve the problem at least on my system before I can help with fixing the inconsistent documentation.

No matter what, it would seem to make sense to point out that there are two potential python kernels that the jupyter notebook can use, and I would argue that installation instructions for running both side-by-side should be included in the core documentation (or at least in the list of available kernels page).

relevant Stack-overflow answers with ambiguous relevance due to big split:

http://stackoverflow.com/questions/30492623/using-both-python-2-x-and-python-3-x-in-ipython-notebook especially the answer: http://stackoverflow.com/a/30493155/1816995 http://stackoverflow.com/questions/32258277/how-can-i-run-python-2-7-3-4-notebooks-side-by-side-in-jupyter-ipython http://stackoverflow.com/questions/29773954/change-ipython-3-for-python-3-kernel-to-python2-for-the-cluster-too http://stackoverflow.com/questions/28831854/how-do-i-add-python3-kernel-to-jupyter-ipython

[takluyver]

To enable a Python 3 kernel when you’re running the server on Python 2, it should be sufficient to do:

pip3 install ipykernel
python3 -m ipykernel.kernelspec --user

And if you run the server on Python 3 and want to add a Python2 kernel, do exactly the same on Python 2:

pip2 install ipykernel
python2 -m ipykernel.kernelspec --user

[takluyver]

At some point in the future, we will probably make the notebook server require Python 3 (but still be able to run Python 2 code in notebooks). This will simplify the explanation a bit, because it will just be ‘setting up a Python 2 kernel’, rather than ‘setting up a Python 2 or 3 kernel, whichever one you’re not running the notebook with’.