Cannot connect to Sparkmagic kernels

Discussed in https://github.com/microsoft/vscode-jupyter/discussions/11125

<div type='discussions-op-text'>

^{Originally posted by jvaesteves August 16, 2022} Hello, I’ve trying to use VSCode to accept the Sparkmagic kernels that I installed on a venv using Poetry, so I can connect to an EMR instance via Livy, but when listing the kernels on the notebook interface, all that appears are Python versions and venvs from my computer. I tried this on PyCharm Pro and works well.

Setup:

• VSCode version: 1.70.1
• Jupyter extension version: v2022.7.1102252217
• Python version: tried 3.10.5 and 3.7.13
• Poetry version: 1.1.14
• Packages: sparkmagic==0.20.0

Steps to reproduce:

SPARKMAGIC_LOCATION=$(pip show sparkmagic | grep Location | cut -d" " -f2)
jupyter nbextension enable --py --sys-prefix widgetsnbextension
jupyter-kernelspec install --user $SPARKMAGIC_LOCATION/sparkmagic/kernels/sparkkernel 
jupyter-kernelspec install --user $SPARKMAGIC_LOCATION/sparkmagic/kernels/pysparkkernel 
jupyter-kernelspec install --user $SPARKMAGIC_LOCATION/sparkmagic/kernels/sparkrkernel 
jupyter serverextension enable --py sparkmagic

What is expected

For PySpark, Spark and SparkR to appear on VSCode kernel list

What is happening

</div>

[jvaesteves]

@DonJayamanne Thanks a lot for the help. With this change, now the PySpark works correctly! 😃

[DonJayamanne]

Closing this issue as its been over 4 weeks, since the information was requested. We’ll be happy to reopen the issue when the requested information has been provided.