Adding extensions to the .NET kernel using nuget packages for F#

What is an extension ? An extension is anything that can be used to modify the behavior of the kernel without necessarily being checked into the kernel code. For example, if we would like the kernel to print “Hello World” after every successful code submission, it can be performed via adding an extension to the kernel.

Possible applications of an extension

1. Adding custom formatters for various types. For example, we can register a formatter for a dictionary which will display the keys and values in the dictionary in a table format with the keys in red color.
2. Support for various visualisation libraries. The Xplot Support that is present right now is done via the kernel extension. The kernel extension when loaded registers a formatter for the XplotChart type.
3. Support for magic commands like %%javascript. Such an extension would possible add a middleware that will read the code submitted and check if the magic command is present and do the evaluation accordingly.

Existing implementation Right now an extension can be added to the kernel using the #extend path_to_dll directive. The problem with this approach is the acquisition story. Once extension author create extensions, how do they make their extension easily available to the notebook users.

Proposed Implementation The extension can be a part of a nuget package. When a user does #r "nuget:MyNugetPackage" we look for a specific path inside the Nuget Package, say /Extensions and try to find all the implementations of IKernelExtension and load them into the kernel.

This approach can also be used to load scripts into the notebook context. Similar to the “Extensions”, there can also be a /Scripts folder inside the nuget package containing the script files. We look for all those script files and pass #load MyScript.csx into the underlying script engine.

Additional questions to be answered The .NET kernel has a CompositeKernel which has subkernels under it. Let’s take the case that the CompositeKernel has two subkernels - C# kernel and F# kernel. How does an extension specify if it is meant for C# kernel, F# kernel or if it is language agnostic and needs to be added to the composite kernel itself?

Also is there a way for the extension to get some piece of information from the kernel, like getting the compilation for the current cell and use that to emit a dll ?

[KevinRansom]

Goodness … this is actually quite hard … the goal is to load up and execute an assembly and all of it’s dependencies in a process.

It should be noted that nuget packages can: 1. Service multiple target frameworks, E.g. net472, net48, netstandard1.6, netstandard2.0, netcoreapp1.0, netcoreapp2.1 … and frameworks as yet unspecified.

2.0 Can target different platforms: I.e. x86, x64, anycpu32bitpreferred, or anycpu

3.0 Can include OS specific native code dlls I.e. MacOS, Linux, Windows.

We will need to be crisp, if we are going to support any of these scenarios with extension loading.
For example F# interactive and compiler supports multiple target frameworks, but only anycpu il assemblies. Therefore we do not differentiate, platform specific libraries and OS specific libraries. (* It is amusing that F# considers word size platform a hangover from msbuild in the old days I suppose *) This behaviour has already shipped for type providers, and has been continues with compiler tools.

Because F# supports multiple platforms the directory structure in a TP looks like:

$PackageRoot\
  lib\
        netstandard2.0\
               MyReferencedAssembly.dll
  typeproviders\
        net462\
               MyReferencedDesignTime.dll
        netstandard2.0\
               MyReferencedDesignTime.dll
        netcoreapp2.1\
               MyReferencedDesignTime.dll

The type provider has a referenced assembly in lib\netstandard2.0. And design time add-ins that can work on net462, netstandard2.0 and netcoreapp2.1 platforms.

However, because of how probing works the same package could be shipped like this: With the tools for a particular library, being packaged as a subdirectory of the library.

I personally think the below should have been an issue, but we shipped this way for a couple of years now. However, I can see why developers kind of like it. It makes debugging simpler, not requiring the developer to actually build the nuget package.

$PackageRoot\
  lib\
        netstandard2.0\
               MyReferencedAssembly.dll
               typeproviders\
                    net462\
                          MyReferencedDesignTime.dll
                    netstandard2.0\
                          MyReferencedDesignTime.dll
                    netcoreapp2.1\
                          MyReferencedDesignTime.dll

FSharp probing is described here: https://github.com/fsharp/fslang-design/blob/master/tooling/FST-1003-loading-type-provider-design-time-components.md

The one change we are going to make for FSharp compiler extensions is that as well as probing the typeproviders folder we will also probe the tools folder.

F# probing requires the presence of the net462, netstandard2.0 or netcoreapp2.1 sub folders.

Having the TFM (for execution) is a two edged sword, it means we can have platform specific TP’s and compiler extensions, however, it means we need to know in advance what TFM (for execution) the executing program supports. It looks to me as if the proposal for .NET kernel is only going to support NetStandard2.0 since you don’t appear to probe for specific tfms.

I like the scripting proposal, although we will probably allow, independent scripts per TFM, enabling the flexibility of having a windows initialization script and a netcoreapp2.1 specific initialization script.

[jonsequitur]

Closing this as the current extension mechanism is simpler, by design. The kernel extension is not language or platform specific. The kernel can implement additional requirements via dependent packages if needed. We can revisit the issue as additional scenarios come up.