CLI-created runtime does not appear even after refresh

Describe the issue After a Kubeflow notebook server has started, we automatically add Elyra runtimes via elyra-metadata install runtimes... (see https://github.com/lehrig/kubeflow-ppc64le-notebook-images/blob/main/post_jupyter_start.sh)

However, this runtime does not appear in Elyra’s runtimes view. Using the “refresh” functionality does not help. Strangely, what does help is to add an arbitrary other runtime using the UI. See this video where I recorded the observed behavior: https://ibm.box.com/shared/static/3lj4iotwcv0653qgzi8e3vfrm7huymmk.mov

To Reproduce Steps to reproduce the behavior:

1. Automatically add a runtime to a new Kubeflow notebook server using Elyra’s CLI
2. Check for available runtimes

Screenshots or log output See this video: https://ibm.box.com/shared/static/3lj4iotwcv0653qgzi8e3vfrm7huymmk.mov

Expected behavior A runtime should directly appear (or at least show up after hitting “refresh”).

Deployment information Describe what you’ve deployed and how:

• Elyra version: 3.8.0
• Operating system: linux (RHEL 8.4 on ppc64le; SELinux enabled)
• Installation source: custom container image (https://github.com/lehrig/kubeflow-ppc64le-notebook-images -> quay.io/ibm/kubeflow-notebook-image-ppc64le:elyra3.8.0-py3.8-tensorflow-cpu2.7.0)
• Deployment type: Kubeflow (v1.5.0) Notebook Server on OpenShift v4.8

Pipeline runtime environment If the issue is related to pipeline execution, identify the environment where the pipeline is executed

• Kubeflow Pipelines (v1.5.0, multi-user auth enabled using OpenShift OAuth)

Runtime configuration settings (see https://github.com/lehrig/kubeflow-ppc64le-notebook-images/blob/main/post_jupyter_start.sh)

[lehrig]

Thanks @ptitzler & @kevin-bates for the call!

Here’s a summary:

• there is no race condition (so the sleep is unimportant); we were able to reproduce the issue also from within the notebook after it started
• the issue can be observed for any kind of metadata (components, runtimes, runtime images, code snippets)
• the issue only happens for the first creation of metadata of a given kind (e.g., the issue occurs for the first creation of a runtime)
• once a second creation of a given kind happens, the issue is actually fixed
• alternatively, creating an arbitrary file inside the metadata folder of a given kind (e.g., ~/.local/share/jupyter/metadata/runtimes) also fixes the issue

I’ve therefore added a hot fix to my script:

touch /home/jovyan/.local/share/jupyter/metadata/runtimes/hotfix2725.fix
rm -f /home/jovyan/.local/share/jupyter/metadata/runtimes/hotfix2725.fix

(see https://github.com/lehrig/kubeflow-ppc64le-notebook-images/commit/eb4eda887e099a35d6fa33dcf6f0df1255207210)

We suspect the underlying reason is the persistent volume mounted into /home/jovyan as being provided by the NFS Provisioner.

[lehrig]

Yes, I would agree that this is not caused by Elyra but rather by the underlying storage system.