Incompatibilities between patch versions when installed in both server and kernel environment

This issue was originally discussed in #34 but I now see this is a distinct issue.

If a jupyterlab user wants to make use of a custom kernel and use ipympl in their notebooks, then ipympl must be installed in both the jupyterlab environment and their kernel environment. If different versions of ipympl are installed in the jupyterlab environment and kernel environments, then sometimes ipympl breaks even for differences in the patch version. If jupyterlab is running within a multi-user environment (jupyterhub) and the user does not control the server, then the user may not know what version of ipympl is installed in the jupyterlab environment. This can make it difficult for users to determine the version of ipympl they need to install in their kernels. Additionally, the kernel environments are likely to break when upgrades occur to ipympl in the jupyterlab environment.

Test setup

The test were run using 6 builds of the following Dockerfile, where LAB_IPYMPL_VERSION modified for each build.

# this is a micromamba:0.19.1 image, using digest instead of tag for reproducibility
FROM mambaorg/micromamba@sha256:1ab2666fcd1559e9865d599433671f97cb735103af9351e5e10a3f0d5b6048c8

# version of ipympl to install into the jupyterlab environment
ARG LAB_IPYMPL_VERSION=0.8.0

# Create conda environments:
# "kernelXXX" environments will be available in jupyter notebooks
# "lab" environment will run the jupyterlab server (created at line 56)

RUN micromamba create --yes --name kernel085 --channel conda-forge \
       ipympl=0.8.5 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel084 --channel conda-forge \
       ipympl=0.8.4 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel083 --channel conda-forge \
       ipympl=0.8.3 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel082 --channel conda-forge \
       ipympl=0.8.2 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel081 --channel conda-forge \
       ipympl=0.8.1 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba create --yes --name kernel080 --channel conda-forge \
       ipympl=0.8.0 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1

# activate the "kernel" environment to be active within in the next RUN command
ARG MAMBA_DOCKERFILE_ACTIVATE=1 

# create a kernel.json file for each environment
ENV ENV_NAME=kernel085
RUN python -m ipykernel install --user --name kernel085 --display-name kernel085
ENV ENV_NAME=kernel084
RUN python -m ipykernel install --user --name kernel084 --display-name kernel084
ENV ENV_NAME=kernel083
RUN python -m ipykernel install --user --name kernel083 --display-name kernel083
ENV ENV_NAME=kernel082
RUN python -m ipykernel install --user --name kernel082 --display-name kernel082
ENV ENV_NAME=kernel081
RUN python -m ipykernel install --user --name kernel081 --display-name kernel081
ENV ENV_NAME=kernel080
RUN python -m ipykernel install --user --name kernel080 --display-name kernel080

RUN micromamba create --yes --name lab --channel conda-forge \
       ipympl=$LAB_IPYMPL_VERSION \
       jupyterlab=3.2.6 \
       jupyterlab_widgets=1.0.2 \
       python=3.10.1 \
       xorg-libxrender=0.9.10 && \
    micromamba clean --all --yes

# activate the "lab" environment for when the CMD executes
ENV ENV_NAME=lab

# RUN jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-matplotlib@0.10.1

EXPOSE 8888
WORKDIR /home/micromamba
CMD ["jupyter", "lab", "--ip=0.0.0.0", "--ServerApp.token=''", "--ServerApp.root_dir=/home/micromamba"]

For each build-kernel pair, I ran the following in a notebook:

import matplotlib.pyplot as plt
%matplotlib widget
fig = plt.figure()
plt.plot([1,2,7])

After selecting the kernel for the next test, I performed a ‘Empty Cache and Hard Reload’ with the the Chrome web browser.

Versions

Here is the version info for the test when ipympl v0.8.0 was installed in the jupyterlab environment:

python -c "import sys; print('\n',sys.version); import ipympl; print('ipympl version:', ipympl.__version__)" && jupyter --version && jupyter nbextension list && jupyter labextension list

 3.10.1 | packaged by conda-forge | (main, Dec 22 2021, 01:39:05) [GCC 9.4.0]
ipympl version: 0.8.0
Selected Jupyter core packages...
IPython          : 8.0.0
ipykernel        : 6.7.0
ipywidgets       : 7.6.5
jupyter_client   : 7.1.0
jupyter_core     : 4.9.1
jupyter_server   : 1.13.2
jupyterlab       : 3.2.6
nbclient         : 0.5.9
nbconvert        : 6.4.0
nbformat         : 5.1.3
notebook         : 6.4.7
qtconsole        : not installed
traitlets        : 5.1.1
Known nbextensions:
  config dir: /opt/conda/envs/lab/etc/jupyter/nbconfig
    notebook section
      jupyter-matplotlib/extension  enabled 
      - Validating: OK
      jupyter-js-widgets/extension  enabled 
      - Validating: OK
JupyterLab v3.2.6
/opt/conda/envs/lab/share/jupyter/labextensions
        jupyter-matplotlib v0.10.0 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)

For the other tests, all these values except for the ipympl version and the jupyter-matplotlib remained constant. jupyter-matplotlib was never directly installed, it always came from the installation of ipympl.

Results

I ran a series of tests to determine which combinations of the ipympl v0.8.x packages are functional when installed in both the jupyterlab environment and the kernel environment. Three different outputs occurred:

1. “model not found” - the text “Error displaying widget: model not found” was displayed
2. “2 button widget”: the following was displayed:
3. “functional” - a full interactive plot with the expected toolbars was displayed

The results are summarized in this table:

Conclusions

ipympl is fragile when installed in kernels in a different environment from the jupyterlab environment. It would be great if in these types of setups, ipympl was compatible with itself across patch versions (ie, within the v0.8.x series).

Thanks!

Sorry this was so long. This module is an important and appreciated part of my workflows. Thanks you!

[ianhi]

I think this will have been fixed going forward by #448 but let’s definitely reopen if we discover that it was not

[ianhi]

It might actually reduce the number of issues “it fails to render in JupyterLab with model not found” that we see popping on all widgets libs.

Agreed I think it would.

We would bump such “model version” only if there is a breaking change in the widget model (e.g. if we rename a widget property in the model). It’s a bit more maintenance pain but it might be worth it.

but I don’t know any widget which does that.

I agree that this is good. But I think the only example is actually ipywidgets itself: https://github.com/jupyter-widgets/ipywidgets/blob/f7c48afb27bd80b14966b862ad03e30c40c7481f/python/ipywidgets/ipywidgets/_version.py#L9-L12