How can a custom deserializer output a native JSON field (record) ?

Here is my situation - I’ve got an event hub feed that contains escaped JSON in a string, i.e. "{\"foo\":\"bar\"}" so I cannot use the built in JSON format. I’ve managed to build a StreamSerializer based on the examples, but I can only yield back simple structures with string or other primitives as fields. I want to return data that will appear as a native JSON output, just like the built-in one does. Here’s what my currently (failing) attempt looks like:

Here’s my structure that’s yielded back from my StreamSerializer<T>:

    public class CustomJsonData
    {
        public string Source { get; set; } = "MySource";
        public string JsonString { get; set; } // works fine
        public JsonDocument Json { get; set; } // fails
    }

The serializer itself:

   public class CustomJsonDeserializer : StreamDeserializer<CustomJsonData>
    {
        // streamingDiagnostics is used to write error to diagnostic logs
        private StreamingDiagnostics streamingDiagnostics;

        // Initializes the operator and provides context that is required for publishing diagnostics
        public override void Initialize(StreamingContext streamingContext)
        {
            this.streamingDiagnostics = streamingContext.Diagnostics;
        }

        // Deserializes a stream into objects of type CustomEvent
        public override IEnumerable<CustomJsonData> Deserialize(Stream stream)
        {
            using (var sr = new StreamReader(stream))
            {
                string line = line = sr.ReadLine();
                JsonDocument json = null;

                try {
                    line = Regex.Unescape(line.Trim('"'));
                    json = JsonDocument.Parse(line);
                } catch (Exception e) {
                    line = e.Message;
                }
                yield return new CustomJsonData() {
                        JsonString = line,
                        Json = json
                };
            }
        }
    }

This works fine dumping out a string, but then it’s just a string field - we don’t get any native JSON support in the Portal UI nor the ability to dot navigate the properties/fields.

So, we could probably use a javascript UDF in the query to eval the string JSON.parse() the string but this seems inefficient and less safe than having our serializer emit native JSON.

The JsonDocument above is from System.Text.Json 5.x (to comply with netstandard 2.0) but we get:

[Info] 2022-01-26 1:33:45 PM : Job Start Successfully !
[Error] 2022-01-26 1:33:55 PM : **System Exception** Could not load file or assembly 'System.Text.Json, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
[Error] 2022-01-26 1:33:55 PM :    at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly)
at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
at System.Runtime.Loader.AssemblyLoadContext.InvokeResolveEvent(ResolveEventHandler eventHandler, RuntimeAssembly assembly, String name)
at System.Runtime.Loader.AssemblyLoadContext.OnAssemblyResolve(RuntimeAssembly assembly, String assemblyFullName)
[Info] Stopped Local Run.
Child process exited with code 4294967295
[Info] Shutdown local credential server http://localhost:8999/

Any tips? Do I need to use a very specific version of System.Text.Json? Do I need a magic attribute somewhere? Do I need to use a differenet base class (i.e. not StreamSerializer?)

Thanks

[Fleid]

Ok I think I got it now, and how I see it is to rather have:

public class CustomJsonData
    {
        public string Source { get; set; } = "MySource";
        public string JsonString { get; set; } // a string
        public dynamic JsonRecord { get; set; } // will be exposed as an ASA record
    }

Where the deserializer can send dynamic objects that will be surfaced in the query as records.

This already works from the the deserialization side of things, I checked. But this type is not supported by ASA. I’m starting a thread with the dev owner to see if we can make that happen (or if there’s another way to go around that).

[oising]

I think maybe you’re overthinking it 😃 All I really would like is a way to return a class/struct with one or more string members that can opt-in to being converted to the ASA Type “Record.” Example:

public class CustomJsonData
    {
        public string Source { get; set; } = "MySource";
        public string JsonString { get; set; } // a string
        [ASARecordType]
        public string Json { get; set; } // will be exposed as an ASA record
    }

Yes, I’d agree that the ASA record type is similar to the DLR’s DynamicMetaObject / C# dynamic but their domain is quite different.