Formalize and extend language transclusion

Elevator Pitch

Allow a user to define rules for finding parts of files in a host language that embed other languages. Additionally support referring to the outer language within the inner language. Language servers, libraries, and other things a user might have installed should be easily and robustly able to ship and activate these on installation.

Motivation

Much like with #190 for #189, there are more languages that embed other languages than we’ll be able to (or want to) define/maintain. The existing list includes things from IPython built-ins, but doesn’t fully capture the scope of other magics, much less what other languages do, especially extremely crazy things like allthekernels, sos or metakernel.

A number of these kinds of things also allow for referring to the outer language, such as {variable} references, up to full-blown templating languages, and having access to completion/diagostics/reference/hover for these nested scopes would be very valuable.

Design Ideas

Add a separate spec to the JSON schema, e.g. transclusions which can be defined/modified without re-compiling lab by traitlets config or entry_points. Move the existing “simple” ones from the hard-coded typescript into defaults on the python side, or simple JSON.

Notional “Simple” Example

LanguageServerManager:
  language_servers: ...
  transclusions:
    ipython-shell-magics:
      host:
        mime_types: 
          - text/x-python
        languages: 
          - python
        file_extensions:
          - .ipy
          - .ipynb
      guest:
        mime_types:
          - application/x-sh
        languages: 
          - sh
        file_extensions:
          - .sh
      patterns:
        line-magic:
          regex: "(?<=\\s)!(.*)$"
          extract: $1
          isolate: true
          host_inclusions:
            variable:
              pattern: "(?<!\\$)\\{([^\\}]+)\\}"
              extract: $1
        cell-magic:
          regex: "^%%bash[^\n]*\n([\n.]+)$"
          extract: $1
          isolate: true
          host_inclusions:
            variable:
              pattern: "(?<!\\$)\\{([^\\}]+)\\}"
              extract: $1

TODO: the ! magic can also be assigned to variables, etc.

x = !ls
# is secretly
x: str = some_function("""ls""")

while

%%bash --bg --proc foo --out foo_out
ls
# is actually
foo: Popen = some_function("""ls""")
foo_out = foo.stdout

More complex things

These would be insufficient for complicated tricks like what’s happening in the some of the r-in-python with declaring variables, and for this case being able to register these from an extension that depends on the to-be ILanguageServerManager (or a dedidcated ITransclusionManager) would be required.

User-serviceable and discoverable

Probably need to be configurable in the frontend, e.g.

• adding/testing new transclusions
• disabling installed ones
• some kind of visual indication of them (gutter?)

[blois]

IPython includes some other magics as well, see https://ipython.readthedocs.io/en/stable/interactive/reference.html#automatic-parentheses-and-quotes

I do think it could be useful to expose an API on the kernel where you can ask for some code to be transformed into the underlying language then receive back the transformed code and a source map of the transformations. This is an established pattern for code transformations that can also be used for lint/format as well as highlighting appropriate locations of code errors from the runtime reported location.

This could even be done extensibly, so nested syntaxes could report their transformations- an embedded SQL query could use it to map SQL errors back to the original source code locations.

[bollwyvl]

Yeah, it’s a rather opinionated project. I don’t mean one should use that repo directly but rather the pattern for publishing it. You might get more leverage from https://github.com/jupyterlab/extension-cookiecutter-ts