Notebook IDE features

What are you trying to do?

The Julynter experiment #378 demonstrated how notebook-specific IDE features could work. The ideas would include:

• “empty cell” diagnostic
• “cells execution not in order” diagnostic (obviously optional)
• “cell with comments only could be a markdown cell” diagnostic
• “remove empty cells” action
• and many others (e.g. “ratio of markdown to code cells”)

How is it done today, and what are the limits of current practice?

• Currently there is no LSP-compliant way to implement such features.
• Users need to install Julynter which does not integrate well with this extension yet.
• The LSP specification has no notebooks support.

What is new in your approach and why do you think it will be successful?

In order to make the language servers (optionally) cell-aware I propose we embrace the jupytext percent format:

# %% Optional title [cell type] key="value"

e.g. # %% for code cells and # %% [markdown] for markdown cell. We could store the cell execution number in metadata (key="value" part). We would allow the user to disable this feature. By embracing the percent format we could achieve some convergence with the Microsoft and PyCharm notebook extensions assuming they would like this idea, with the former important if we want to get this into LSP standard.

As the comments differ in syntax between languages we should probably closely collaborate with jupytext (@mwouts) as they already handle this problem for a numer of languages.

Having the right format at the language server we would then:

• prepare one or two extensions to the existing language servers to demonstrate how one would interpret the comments in static analysis and emit appropriate diagnostics.
• encourage community to provide more such extensions
• having a proof of concept maybe try to make it part of the LSP protocol
• eventually consider packaging language servers with these extensions as defaults for jupyter-lsp (e.g. we could package version-frozen pyls with notebook features plugins as pyls-notebook or ipython-langauge-server)

On the frontend we would need to:

• get the comment characters specific to the language:
  • provided by backend from jupytext if installed, or
  • use the ones from CodeMirror if not know for given language from backend
• render the diagnostics on cells, for example by highlighting cells in the appropriate color; here the lessons learned by Julynter experiment (@JoaoFelipe et al) would be extremely valuable.

What are the risks?

It would be a shame if someone else develops a completely different standard without reaching out to us and we would have more fragmentation. Lets try to keep developers of other tools in the loop once the idea matures a bit and announce it on the appropriate forums of standard authorities.

How long will it take?

I have no spare resources to allocate to this myself as those are low priority to my research. It would be largely in a planning phase there until the next holiday where I will possibly have more time to implement this, or someone else who is interested picks this up earlier (contributions welcome).

[agoose77]

Having in-virtual-document directives would become problematic if these transforms become finer grained (e.g. per token).

Could you expand on this one?

@krassowski I think perhaps I’m over-thinking this. I was conceiving of “transforms” that are not simple transclusions, and add tokens to the document. This isn’t actually something any logical use of LSP should support on reflection.

I used this strategy and originally tought it is a good idea, but it does not work with complex refactor/reformatting operations (because as mentioned above the language server is free to rearrange the inlined document in any way it wants without providing a map of which lines moved where).

Right, so you want to keep some in-document anchors for which there will already be some structural stability (i.e. comments in most (all?) languages). The nice thing about doing this in a “Notebook adaptor” backend is that this is all just implementation details 😃

[krassowski]

Having in-virtual-document directives would become problematic if these transforms become finer grained (e.g. per token).

Could you expand on this one?

can’t we maintain that information separately?

My hunch is that it won’t be possible because the language server should be free to rearrange the code in any way it wants (reformat, move some lines to another editor, add some lines, etc) and it is free not tell us what it intends to do, but often it will just give us an end result (reformatted source file). Without having e.g. comments indicating cell breaks it becomes effectively impossible to map the modified source code back to the notebook cells (if there are no comments indicating where each cell begins).

For the existing functionality, a view would just need to maintain an internal one-to-one (non-surjective) map from the virtual document to a given cell location in the original notebook. Assuming we only needed line numbers for simplicity, this would just be

I used this strategy and originally tought it is a good idea, but it does not work with complex refactor/reformatting operations (because as mentioned above the language server is free to rearrange the inlined document in any way it wants without providing a map of which lines moved where).