[BUG]: .Net Interactive: UDF use deprecated BinaryFormatter, fail in VS Code

Describe the bug UDFs are serialized using BinaryFormatter, which is deprecated for security reasons.

In environments where BinaryFormatter is already disabled, this causes UDFs to fail with a NotSupportedException.

To Reproduce

Steps to reproduce the behavior:

1. Use Visual Studio Code with the .Net Interactive Notebooks extension
2. Follow the steps in UDFs in .NET Interactive
3. Executing cell #3 (which contains the definition of the UDF) fails with:

Error: System.NotSupportedException: BinaryFormatter serialization and deserialization are disabled within this application. See https://aka.ms/binaryformatter for more information.
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph)
at Microsoft.Spark.Utils.CommandSerDe.Serialize(Delegate func, SerializedMode deserializerMode, SerializedMode serializerMode)
at Microsoft.Spark.Sql.Functions.CreateUdf(String name, Delegate execute, PythonEvalType evalType, String returnType)
at Microsoft.Spark.Sql.Functions.CreateUdf[TResult](String name, Delegate execute, PythonEvalType evalType)
at Microsoft.Spark.Sql.Functions.CreateUdf[TResult](String name, Delegate execute)
at Microsoft.Spark.Sql.Functions.Udf[T,TResult](Func`2 udf)
at Submission#11.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)

Expected behavior

• Should be able to use UDFs
• The example in UDFs in .NET Interactive should work
• UDFs should not use BinaryFormatter

Screenshots NA

Desktop (please complete the following information):

• OS: macOS 10.15.7
• .NET 5 (5.0.100)
• Apache Spark 3.0.1
• .NET for Apache Spark v1.0.0 (netcoreapp3.1, macOS)
• VS Code:
• Version: 1.53.0-insider
  • Commit: 96b426ef1da0f0a51ce9ef1931cd1fd2322547e4
  • Date: 2020-12-14T05:49:45.285Z
  • Electron: 11.0.3
  • Chrome: 87.0.4280.67
  • Node.js: 12.18.3
  • V8: 8.7.220.25-electron.0
  • OS: Darwin x64 19.6.0
• .Net Interactive Notebooks v1.0.160901

Additional context NA

[imback82]

Thanks @oefe for reporting this. We will move away from BinaryFormatter from upcoming releases.

[GeorgeS2019]

We will move away from BinaryFormatter from upcoming releases.

@imback82 Is there a solution to this, now is 2023?