Missing kernelspec on new notebooks created in VSCode

Describe the issue Notebooks created in VSCode don’t have a kernel spec until you run them, which causes issues when trying to run them with Elyra.

To Reproduce Steps to reproduce the behavior:

1. Create and open an .ipynb file in VSCode
2. Add some content to it
3. Save the notebook file
4. Open the notebook Jupyter Lab
5. Add the notebook to an Elyra Pipeline
6. Set a runtime image for the node
7. Submit notebook for local execution
8. Get error KeyError: 'kernelspec'

Screenshots or log output

Expected behavior Either a better error message with steps to resolve the issue, or just magically run somehow. (This can be fixed by opening the notebook in Jupyter Lab and re-saving the file or by running the notebook at least once in VSCode)

Deployment information Describe what you’ve deployed and how:

• Elyra version: 1.5.3
• Operating system: macos
• Installation source: from source
• Deployment type: local installation

Pipeline runtime environment If the issue is related to pipeline execution, identify the environment where the pipeline is executed

• Local execution

Notebook source:

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": 3
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"hello world\")"
   ]
  }
 ]
}

[kevin-bates]

If we’re simply trying to improve the user experience with a better message or possibly interrogating the language-info, then I can take this. The crux of this issue is that papermill makes an assumption that the kernelspec will be in the NB metadata. Since they allow a kernel_name to override that, I’m sure the idea is that, in those cases, they expect the user (vscode) to provide the kernel name.

I think the right thing to do here is add best-fit logic like we do in kfp, but extend it to include checks against the language-info (which should also apply to kfp).

I don’t think we want to catch KeyError from papermill, but I suppose that could be a fallback should we run into issues with language-info code.

So the idea would be - since this is local processing - check the NB metadata, if a kernelspec exists, done (let default scenario happen - even if that particular kernel is not defined, the error message will be appropriate), else, look at language-info to determine language, then hit the kernelspec manager to find matching language. We should probably check if the “default” kernel is also a language match and let that override any (if it matches the lang). Once found, set the kernel_name to that when calling Papermill.

I’ll go ahead and assign this to me.

[kevin-bates]

Ok - per a side-bar discussion, the server will detect missing kernelspec information. If the information is present, but the kernel is not available, we’ll use the language indicator and select the first kernel found that corresponds to that language (which I believe is the case today). If no language (or kernelspec information) is present, we’ll use the default kernel name configured on the notebook server’s KernelManager. A warning message should be produced indicating this defaulting behavior has taken place.

It also sounds like VSCode “insiders” will be updating the notebook with kernelspec information on first use. However, the server changes described above are good to have in place anyway.

Thanks @bourdakos1!