pyspark kernel created using sparkmagic is not showing in the kernel list of jupyter extension in vs code

Environment data

• VS Code version: 1.62.2
• Jupyter Extension version (available under the Extensions sidebar): v 2021.10.110
• Python Extension version (available under the Extensions sidebar): v. 2021.10.11422169775
• OS (Windows | Mac | Linux distro) and version: Windows 10 Enterprise 20H2 build 19042.1288
• Python and/or Anaconda version: python 3.8.8, Anaconda 3 (64 bits)
• Type of virtual environment used (N/A | venv | virtualenv | conda | …): NA
• Jupyter server running: Local | Remote | local

Expected behaviour

All kernel visible/working in Conda Jupyter Notebook should be the same in VS code jupyter extension

Actual behaviour

pyspark kernel installed using sparkmagic did not show in vs code jupyter extension kernel list, even it worked well with Conda Jupyter Notebook and it showed with command of jupyter kernelspec list.

Steps to reproduce:

1. Install sparkmagic extension (instruction: https://github.com/jupyter-incubator/sparkmagic)
2. test in Jupyter Notebook, and it should be available in the kernel list
3. Open vs code, choose kernel, it is not showing in the list.

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. XXX

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

XXX

[jpmaterial]

If anyone is wondering where to find the kernelspec: jupyter kernelspec list gives a list of all directories specifying kernels, which will contain kernel.json files.

[fkamp]

I’m closing this issue as we’ve resolved the issues around displaying custom kernels (in the latest dev branch, soon to be released). If the issue persists, please feel free to re-open this issue or create a new one and we’ll be happy to look into them.

@DonJayamanne, I still faced this issue in this version: VSCode 1.71.2 Jupyter Extension v2022.8.1002431955 Python Extension v2022.14.0

This was missing until I updated “python” to the full path

{
  "argv": [
    "python",
    "-m",
    "aws_glue_interactive_sessions_kernel.glue_pyspark.GlueKernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Glue PySpark",
  "language": "python"
}

Interestingly, this works fine without it:

{
  "argv": [
    "python",
    "-m",
    "aws_glue_interactive_sessions_kernel.glue_spark.GlueKernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Glue Spark",
  "language": "scala"
}