Bidirectional communication between kernel and notebook UI

We want to be able to create ‘live’ UI elements in a notebook that can be updated by ongoing activity elsewhere in the notebook. Today, we can produce HTML output and client-side JavaScript to enable interactivity, but there is currently no built-in mechanism to connect this in any way to code running in the kernel.

To support this, we would like it to be possible to open channels of communication by which code running inside the kernel (either code cells, or extensions or other components loaded by those cells) is able to send messages to and receive messages from JavaScript (or TypeScript) code running in client hosting the notebook.

It’s not completely impossible to achieve this today, but it’s messy and limited. We have built a prototype that provides limited support for such connectivity. It works by running a simple web server inside the kernel process, and then connecting to this from JavaScript code on the client side. UI to kernel communication can be achieved with fetch operations from the client, and we used SignalR to enable the kernel to send messages to the client code. We have been able to demonstrate the kind of ‘live’ notebook cells we have in mind. However, there are a number of issues with this technique.

For example, currently it only works when the kernel process is local; we have not yet made it work for remote scenarios. We have only attempted this in Visual Studio Code, and not yet other hosts. Nor have we attempted to get it working in CodeSpaces yet. In principle, these are all just challenges which could be overcome with sufficient engineering effort. However, it is likely to be fragile, because there may be scenarios in which our chosen communication mechanism might not be open to us. In any case, it duplicates work already being done by .NET Interactive.

.NET Interactive already maintains a channel enabling communication between the UI process and the kernel process. (As it happens, it even uses SignalR today.) Instead of building a parallel mechanism to this, it would be far more robust and reliable to be able to piggy back on the channel that .NET Interactive has necessarily already established.

We are requesting that a message passing service be layered on top of this.

In the kernel, the existing Microsoft.DotNet.Interactive API’s Kernel type would offer a method (say, GetChannel) enabling an object representing a named channel to be acquired. It might look something like this:

public interface IKernelChannel
{
    IObservable<KernelChannelMessage> Messages { get; }
    // TODO: should this be async, enabling callers to know the message has been delivered?
    void SendMessage(KernelChannelMessage message);
}

A similar facility would exist for client-side code. JavaScript in the notebook would be able to obtain a very similar sort of service. The KernelClient interface would offer a method (say, getChannel) which would return an object providing the same two services: an observable source of messages, and a method by which a message can be sent.

When the UI-side code sends a message, this would result in the Messages observable in the kernel reporting the message. Likewise when the kernel invokes SendMessage it would cause the observable source on the UI side to report the message.

Issues to be resolved include:

• What format would the message data use? JSON strings are one obvious possibility but not the only one
• What other metadata would we want in the KernelChannelMessage? Probably some sort of message type indication; anything else? (E.g., message id? In reply to (correlation) id?)
• Would we want to provide support for more structured idioms, e.g., an RPC-like (request/response) mechanism built on top of this underlying duplex channel? Or pluggable support for other protocols on top of the channel?

[jonsequitur]

appmessage: (currently also called “messages” in that spike): an application-defined unit of information comprising a label and a payload, delivered by executing a particular command; the command has no behaviour besides making the appmessage available for the app

Is this second level of indirection necessary? My impression is it would lead to a duplication of serialization, pattern matching, and dispatch implementations, one for the command and event types and another for the various message types. Right now we have, for example:

• SubmitCode
• RequestCompletions
• etc.

Each has a corresponding JSON representation which, with our standard envelope, looks like this:

{
    "token": "the-token",
    "commandType": "SubmitCode",
    "command": {
        "code": "123",
        "submissionType": 0,
        "targetKernelName": "csharp"
    }
}

An approach we’ve discussed is to allow this list to become extensible, so people could introduce custom command types, e.g.:

{
    "token": "the-token",
    "commandType": "MyCustomCommandType",
    "command": {
        "property1": "123",
        "property2": 456
    }
}

It would be a peer to the existing commands and use the same code paths for serialization and so on. It can be mapped to strongly-typed commands, but of course you could still create a looser approach within your own custom messages.

But it would be nice to avoid creating messages like the following, where JSON contains more encoded JSON, and deserialization has to happen twice:

{
    "token": "the-token",
    "commandType": "SendMessage",
    "command": {
        "Type": "MyCustomMessage",
        "Content": "{\"property1\":\"123\",\"property2\":456}"
    }
}

[jonsequitur]

I think we’re in a good place to move forward with one or more PRs. I’m happy to work incrementally. If you want to include documentation in the PR that’s definitely helpful but I tend to focus more on tests as documentation of the intent and API at first, so I’m also happy to wait until the complete feature is ready before documenting it.

Also, there’s going to be some churn in the Microsoft.DotNet.Interactive.Http project and some new infrastructure for setting up routes that I think will be very helpful for this. Because there’s a high likelihood for overlap there, it might be useful for us to focus first on the changes in the Microsoft.DotNet.Interactive library and work our way out. A quick punch list:

• Sending and handling custom commands and events at the Kernel level. (#895)
• A mechanism for enforcing serialization rules for custom commands and events. (Right now we use Assent tests but this really only works for the types we know of at compile time.) (#896)

FYI @halter73