Language Server Kernel

Elevator Pitch

Write a wrapper kernel that can act as the proxy between a(ll) language server(s), replacing custom websocket connection wrangling.

Motivation

Extending on #184, (and thinking about #237 which i have been able to reproduce when binder is getting thrashed), and some broader comments in the community… Lab 2 reworked sessions stuff so it is/tries to be even more robust in the face of degraded connections. Trying to get (and maintain) that level of functionality will be hard for us.

Maximizing reuse of the kernel infrastructure would:

• get users all the buffering/reconnection/starting/stopping basically for free
• give us access to better tooling for kernel messages than we’ll be able to build for our custom websocket transport
• obviate the need for some of the backend complexity (at least in the manager)
  • though we’d need new complexity to handle dealing with “plain” kernels for the listeners, etc… unless all of that functionality could be moved into the kernel itself
• obviate the need for the clunky (and huge) webpack build of lsp-ws-connection

Design Ideas

The LanguageServerKernel would be an ipykernel.kernelbase.Kernel subclass. It would (awkwardly) show up in the kernel listing (though we could we hide it with css… but it might be cool to interact with it, don’t know).

Since we’d still need the serverextension enabled (somehow #257), it could enable the kernelspec pretty directly, rather than needing another file, as something like ilsp.

The frontend would start a KernelSession for ilsp. When it came back with its kernel_info, it might be able to give us all of the server listing stuff, and we’d be able to start talking to it. Presumably the starting/stopping of language servers could be some pithy JSON-schema-constrained language, rather than being “full” python.

Much of the existing stdio machinery would be reused, but would have basically zero chance of hanging the notebook server process, which is A Good Thing. The listener API would be… more complicated to maintain, but it can’t all be roses.

As for actually integrating with the kernel messaging protocol: custom comms, used by widgets and vdom, already give us a way to add this kind of feature in a way that could be used by other kernels in the future, provided they had some way to “opt in” to receiving LSP messages.

[bollwyvl]

Quick update: I’ve taken the first steps at creating such a kernel:

https://github.com/bollwyvl/jupyterlab-lsp/blob/add-language-server-kernel/py_src/jupyter_lsp/kernel/kernel.py

The good parts:

• the LanguageServerManager basically Just Works, and is delivering messages
• aside from having an lsp_manager, it quacks like a normal ipython kernel
  • you can open up a notebook or a console and connect it to the kernel the extension uses and see what’s going on it there… much better than printing stuff inside a websocket

What it does do, thus far:

• instead of bespoke websockets, uses Jupyter comms (much like widgets, bokeh, vdom, and a few other things)
• handles fetching available sessions

What it doesn’t do, yet:

• actually service the ConnectionManager
• read everything off config properly (much of that was being handled by the notebook server config)

It’s going to take a while, but this is probably going to work. 🤞

[bollwyvl]

how debugger handles files

As mentioned elsewhere, DAP and LSP don’t agree exactly on the definition of a file… 🤷?

and comms

Supporting the debugger the way it was done is via a change to the kernel message protocol… might have been a new channel (vs e.g. iopub), which is very seldom changed, and few kernels support it, while most “full” kernels that might want to support LSP already support comms, because widgets are 💰 .

In light of the above, it doesn’t bother with comms at all, to my knowledge. xeus-python does fully support comms, but, for example, xeus-robot or sqlite does not.

For our purposes, while supporting languages that do have kernels contributes a lot of complexity, we’re dealing with a lot of languages that don’t have kernels, so we’re still going to need something for those language servers. I am still of the mind that the comm-based approach fits more cleanly into the existing architecture and ecosystem vs. our existing custom websocket/REST stuff. So having a good spec for what a LSP-over-comm.

October is very busy for me so well be back at it in November.

No worries, I think once Lab 3 drops and has a few shake-out releases, we’re really going to want to get on that bus. pip install jupyterlab-lsp, and getting the labextensions (as, e.g. jupyterlab-lsp-labextension), and serverextension jupyter_lsp (or kernel as ipylsp or , eventually,xeus-lsp) is a lot nicer. For things that can be in the browser, my real hope is we’ll be able to support pip install jupyterlab-lsp-<lang>-labextension, as well, to really get nodejs out of the picture for users that don’t want it (me, and most package managers).