Learning FP using Language-Ext / A usecase for Free Monad

I’ve just started on track to be reformed programmer (Louthism!) and eager to handle this usecase functionally in general and using Language-ext in particular.

Suppose we have following source data in json format:

[
  {"Type":"Asset","Id":"Account Recievable","Amount":50000},
  {"Type":"Asset","Id":"Cash In Hand","Amount":10000},
  {"Type":"Asset","Id":"Bank ABC","Amount":100000},
  {"Type":"Expense","Id":"Salary","Amount":30000},
  {"Type":"Expense","Id":"Office Rent","Amount":6000},
  {"Type":"Expense","Id":"Utilities","Amount":4000},
]

And want output data computed based on source data:

[
  {"Type":"Computed","Id":"Total Assets","Amount":160000},
  {"Type":"Computed","Id":"Total Expenses","Amount":40000},
  {"Type":"Computed","Id":"Balance","Amount":120000},
]

We want to provide with ability to declare what other basic values on one wants to compute.

To compute desired output from source data, following declaration/specfication in json format might be a reasonable start:

[
  {
    "Type":"Compute",
    "Id":"Total Assets",
    "Operator":"Sum",
    "SourceType":"Asset",
    "Remarks":"Sum all with type Assets"
  },
  {
    "Type":"Compute",
    "Id":"Total Expenses",
    "Operator":"Sum",
    "SourceType":"Expense",
    "Remarks":"Sum all with type Expense"
  },
  {
    "Type":"Compute",
    "Id":"Balance",
    "Operator":"Expression",
    "SourceType":"{Total Assets} - {Total Expression}",
    "Remarks":"Compute Balance"
  },
  {
    "Type":"Print",
    "Remarks":"Print messages after substituting computed Ids in each message format string",
    "Messages":[
      "Total Assests : {Total Assets}",
      "Total Assests : {Total Expenses}",
      "Balance (Total Assets - Total Expenses): {Balance}"
    ] 
  }

]

After digging through Langugage-ext related resources (wiki, forums, samples, tests, etc.), it seems this problem can be modeled similar to BankingAppSample with BankFree free Monad.

Essentially we need to write “Interpreter” for the Json Specfication/Declaration to transform input data source to desired output.

Need some help on whether am thinking in right direction and any resource that can help addressing this usecase or to the extent where someone willing to give it a shot.

(Note: On advice of @louthy, this post is copied from gitter post on 6th Apr 2018.)

[louthy]

@imranypatel So, if you want to do this, as well as it can be done - and learn the functional thing on the way. Then (based on what you’re asking for) I would do the following (btw, I have implemented this exact setup twice in the past few weeks, so I know it will work, but it’s going to be a steep learning curve).

Here’s a working gist of what I’m going to talk you through:

https://gist.github.com/louthy/b3a40c7f5f1bc1a8431c770230800ecb

Firstly the idea of building a Free monad is exactly how this will work well. Note however that lang-ext doesn’t have generalised free-monad support (one day I’ll work out how to generalise it, but it isn’t there now). So, each time you want to use a technique like this you have to do the plumbing yourself.

Next, to do the expression stuff, then you should use LanguageExt.Parsec - this is what it’s built to do. It can build a mini-DSL language that you can use to make either a simple expression system right up to a fully fledged language.

The parser you build with LanguageExt.Parsec should ‘compile’ to the same Free monad. The Free monad type will need to have cases to handle expressions. This will allow your entire process to be one computation that both runs the operations but also runs the expressions.

Once you have the Free monad definition you need to define the Interpreter to run it.

So, first I’ll start off with the Free monad definition. The first step is to create ‘an algebra’ for all the operations you want to perform. What this means is a ‘discriminated union’ the describes every operation. C# doesn’t support discriminated unions, so we use inheritance.

Below is the core type Transform<A> and its ‘cases’:

    public class Transform<A>
    {
        public class Return : Transform<A>
        {
            public readonly A Value;
            public Return(A value) => Value = value;
        }

        public class Fail : Transform<A>
        {
            public readonly string Value;
            public Fail(string value) => Value = value;
        }

        public class AllRows : Transform<A>
        {
            public readonly Func<Seq<AccountingRow>, Transform<A>> Next;
            public AllRows(Func<Seq<AccountingRow>, Transform<A>> next)
            {
                Next = next;
            }
        }

        public class FilterRows : Transform<A>
        {
            public readonly string Value;
            public readonly Func<Seq<AccountingRow>, Transform<A>> Next;
            public FilterRows(string value, Func<Seq<AccountingRow>, Transform<A>> next)
            {
                Value = value;
                Next = next;
            }
        }

        public class Log : Transform<A>
        {
            public readonly string Value;
            public readonly Func<Unit, Transform<A>> Next;
            public Log(string value, Func<Unit, Transform<A>> next)
            {
                Value = value;
                Next = next;
            }
        }

        public class SetValue : Transform<A>
        {
            public readonly string Name;
            public readonly object Value;
            public readonly Func<Unit, Transform<A>> Next;
            public SetValue(string name, object value, Func<Unit, Transform<A>> next)
            {
                Name = name;
                Value = value;
                Next = next;
            }
        }

        public class GetValue : Transform<A>
        {
            public readonly string Name;
            public readonly Func<object, Transform<A>> Next;
            public GetValue(string name, Func<object, Transform<A>> next)
            {
                Name = name;
                Next = next;
            }
        }

        public class Compute : Transform<A>
        {
            public readonly ComputeOperation Operation;
            public readonly Func<Unit, Transform<A>> Next;
            public readonly SourceType SourceType;
            public Compute(ComputeOperation operation, SourceType sourceType, Func<Unit, Transform<A>> next)
            {
                Operation = operation;
                SourceType = sourceType;
                Next = next;
            }
        }

        public class Print : Transform<A>
        {
            public readonly PrintOperation Operation;
            public readonly Seq<string> Messages;
            public readonly Func<Unit, Transform<A>> Next;
            public Print(PrintOperation operation, Seq<string> messages, Func<Unit, Transform<A>> next)
            {
                Operation = operation;
                Messages = messages;
                Next = next;
            }
        }
    }

Note how most of the classes have a Next field which is a Func. That is how the operations are chained together. A bit like continuations. Only Return and Fail don’t have Next, and that’s because they terminate the computation.

For the type to become a monad you need to define the standard monad operations:

    public static class TransformLINQ
    {
        public static Transform<B> Bind<A, B>(this Transform<A> ma, Func<A, Transform<B>> f)
        {
            switch (ma)
            {
                case Transform<A>.Return item: return f(item.Value);
                case Transform<A>.Fail item: return new Transform<B>.Fail(item.Value);
                case Transform<A>.Log item: return new Transform<B>.Log(item.Value, n => item.Next(n).Bind(f));
                case Transform<A>.SetValue item: return new Transform<B>.SetValue(item.Name, item.Value, n => item.Next(n).Bind(f));
                case Transform<A>.GetValue item: return new Transform<B>.GetValue(item.Name, n => item.Next(n).Bind(f));
                case Transform<A>.Compute item: return new Transform<B>.Compute(item.Operation, item.SourceType, n => item.Next(n).Bind(f));
                case Transform<A>.Print item: return new Transform<B>.Print(item.Operation, item.Messages, n => item.Next(n).Bind(f));
                case Transform<A>.AllRows item: return new Transform<B>.AllRows(n => item.Next(n).Bind(f));
                case Transform<A>.FilterRows item: return new Transform<B>.FilterRows(item.Value, n => item.Next(n).Bind(f));
                default: throw new NotImplementedException();
            }
        }

        public static Transform<B> Map<A, B>(this Transform<A> ma, Func<A, B> f) =>
            ma.Bind(x => Transform.Return(f(x)));

        public static Transform<B> Select<A, B>(this Transform<A> ma, Func<A, B> f) =>
            ma.Bind(x => Transform.Return(f(x)));

        public static Transform<B> SelectMany<A, B>(this Transform<A> ma, Func<A, Transform<B>> f) =>
            ma.Bind(f);

        public static Transform<C> SelectMany<A, B, C>(this Transform<A> ma, Func<A, Transform<B>> bind, Func<A, B, C> project) =>
            ma.Bind(a => bind(a).Map(b => project(a,b)));
    }

NOTE: Every time you add a new case to the type you need to add it to the switch in the Bind function. The important thing here is that each one is basically the same, it’s recreating the same case type, but with a new Next function, and each time you do it it’s the same. So it’s quite easy to do.

What this does is allow non-Transform<A> behaviours to be injected without defining new cases. This enables LINQ operations, etc.

Now, for convenience we create a static type called Transform which is a factory type for creating new Transform<A> types. The important thing with these factory functions is we don’t need to supply Next, it uses Return as the default Next function.

    public static class Transform
    {
        public static Transform<A> Return<A>(A value) => 
            new Transform<A>.Return(value);

        public static Transform<A> Fail<A>(string value) => 
            new Transform<A>.Fail(value);

        public static Transform<Unit> Log(string value) =>
            new Transform<Unit>.Log(value, Return);

        public static Transform<Seq<AccountingRow>> AllRows =
            new Transform<Seq<AccountingRow>>.AllRows(Return);

        public static Transform<Seq<AccountingRow>> FilterRows(string value) =>
            new Transform<Seq<AccountingRow>>.FilterRows(value, Return);

        public static Transform<Unit> SetValue(string name, object value) =>
            new Transform<Unit>.SetValue(name, value, Return);

        public static Transform<object> GetValue(string name) =>
            new Transform<object>.GetValue(name, Return);

        public static Transform<Unit> Compute(ComputeOperation op)
        {
            var sourceType = from expr in Parse(op.SourceType)
                             from type in op.Operator == "Expression"
                                  ? from value in Compiler.Compile<object>(expr)
                                    select new ValueSourceType(value) as SourceType
                                  : from rows in FilterRowsBy(expr)
                                    select new RowsSourceType(rows) as SourceType
                             select type;

            return from type in sourceType
                   from compute in new Transform<Unit>.Compute(op, type, Return)
                   select compute;
        }

        static Transform<Seq<AccountingRow>> FilterRowsBy(Scripting.Expr expr) =>
            expr is Scripting.IdentifierExpr ident
                ? FilterRows(ident.Value)
                : Fail<Seq<AccountingRow>>($"Invalid SourceType: {expr} for non-Expression operation");

        public static Transform<Unit> Print(PrintOperation op)
        {
            var messages = op.Messages
                             .Map(x => $"\"{x}\"")
                             .Map(Parse)
                             .Map(msg => from e in msg
                                         from r in Compiler.Compile<string>(e)
                                         select r);

            return from msgs in Sequence(messages)
                   from compute in new Transform<Unit>.Print(op, msgs, Return)
                   select compute;
        }

        static Transform<Scripting.Expr> Parse(string script) =>
            Scripting.Parse(script).Match(
                Right: e => Return(e),
                Left: er => Fail<Scripting.Expr>(er + " at " + script));

        static Transform<Seq<A>> Sequence<A>(Seq<Transform<A>> seq) =>
            seq.IsEmpty
                ? Return(Seq<A>())
                : from x in seq.Head
                  from xs in Sequence(seq.Tail)
                  select Prelude.Cons(x, xs);
    }

Now, some of those functions are more complex than just creating new Transform<A> types, and that’s because they support scripting (SourceType in Compute and Messages in Print). What they do is the parse the text first into a Scripting.Expr, then they compile the expression into a Transform<A>. That means the scripts are running in the same context as the bulk operations.

To achieve this I build a language parser using LanguageExt.Parsec. It does a lot for free, you’ve almost got an entire language here. I won’t walk you through all of that, but essentially the Scripting type builds the core parsers, an operator precedence and associativity table, and then calls buildExpressionParser which does the hard work of making a robust expression parser. So, calling Scripting.Parse(text) will give you a Scripting.Expr which is an abstract syntax tree.

You then pass the expression to the Compiler.Compile function which walks the syntax tree building a single Transform<A>.

What’s nice about this approach is you will get proper contextual errors out of the thing if there are syntax errors, or typos, etc. And it can work in the same context as the bulk computation. That means it can access the rows, or whatever.

So, now we have a way of manually creating a Transform<A>, we need a way of creating it from the ‘schema’ JSON. Now, I’m not going to do the work of loading JSON for you, so take a look at the Data type:

    public static class Data
    {
        public static readonly Seq<AccountingRow> Rows = Seq(
            new AccountingRow("Asset", "Account Recievable", 50000),
            new AccountingRow("Asset", "Cash In Hand", 10000),
            new AccountingRow("Asset", "Bank ABC", 100000),
            new AccountingRow("Expense", "Salary", 30000),
            new AccountingRow("Expense", "Office Rent", 6000),
            new AccountingRow("Expense", "Utilities", 4000));

        public static readonly Seq<IOperation> OperationsToPerform = Seq(
           ComputeOperation.New("TotalAssets", "Sum", "Asset", "Sum all with type Assets"),
           ComputeOperation.New("TotalExpenses", "Sum", "Expense", "Sum all with type Assets"),
           ComputeOperation.New("Balance", "Expression", "TotalAssets - TotalExpenses", "Compute Balance"),
           PrintOperation.New("Print messages after substituting computed Ids in each message format string",
                Seq(
                    "Total Assets : {TotalAssets}",
                    "Total Expenses : {TotalExpenses}",
                    "Balance (Total Assets - Total Expenses): {Balance}")));

        public static Transform<Unit> Load(Seq<IOperation> operations) =>
            operations.IsEmpty 
                ? Transform.Return(unit)
                : from head in 
                      operations.Head is ComputeOperation c ? Transform.Compute(c)
                    : operations.Head is PrintOperation p   ? Transform.Print(p)
                    : Transform.Fail<Unit>("Invalid operation")
                  from tail in Load(operations.Tail)
                  select tail;
    }

It has your original test data in it - but also a simple Load function that takes a sequence of IOperation values and then compiles it into a Transform<Unit>. What’s good about this is the calls to Transform.Compute(c) and Transform.Print(p) will also be compiling the scripts.

Now, that’s the raw type side of it done. We need to implement an interpreter to run the Transform<A> DSL.

First off we need some state that will be managed by the interpreter as its running:

    public class InterpreterState
    {
        public readonly Map<string, object> Vars;
        public readonly Seq<AccountingRow> Rows;
        public readonly Subject<string> Output;

        public readonly static InterpreterState Empty = new InterpreterState(default, Seq<AccountingRow>(), new Subject<string>());

        public InterpreterState(Map<string, object> vars, Seq<AccountingRow> rows, Subject<string> output)
        {
            Vars = vars;
            Rows = rows;
            Output = output;
        }
        public InterpreterState SetVar(string name, object value) =>
            With(Vars: Vars.AddOrUpdate(name, value));

        public Option<object> GetVar(string name) =>
            Vars.Find(name);

        public InterpreterState With(
            Map<string, object>? Vars = null,
            Seq<AccountingRow> Rows = null, 
            Subject<string> Output = null
            ) =>
            new InterpreterState(
                Vars ?? this.Vars,
                Rows ?? this.Rows,
                Output ?? this.Output
                );
    }

It has three fields:

• Vars - this will be used to store and load the variables defined during the computation (like TotalAssets, etc.)
• Rows - this will get the raw data-set that needs to be processed
• Output - this is an Rx stream that you can subscribe to to get information out of the system. You don’t have to do it this way, the state could keep a running total of data sets or whatever.

Now the InterpreterState type is defined we need to implement the interpreter itself:

    public static class Interpreter
    {
        public static Either<string, A> Interpret<A>(Transform<A> ma, InterpreterState state)
        {
            switch(ma)
            {
                case Transform<A>.Return r: return Right(r.Value);
                case Transform<A>.Fail f: return Left(f.Value);
                case Transform<A>.Log m:
                    state.Output.OnNext(m.Value);
                    return Interpret(m.Next(unit), state);
                case Transform<A>.GetValue op:
                    return state.GetVar(op.Name)
                         .Match(Some: v  => Interpret(op.Next(v), state),
                                None: () => Left($"Unknown value: {op.Name}"));
                case Transform<A>.SetValue op:
                    return Interpret(op.Next(unit), state.SetVar(op.Name, op.Value));
                case Transform<A>.Compute op:
                    return Compute(op, state);
                case Transform<A>.Print op:
                    return Print(op, state);
                case Transform<A>.FilterRows op:
                    return FilterRows(op, state);
                default:
                    throw new NotImplementedException(ma.GetType().Name);
            }
        }
    }

This is just a big switch statement, note how Return and Fail don’t call Interpret like all the others. Interpret is a recursive function that gets passed the result of the last operation so it can run its current operation. This walks the Transform<A> DSL - essentially running it.

There are three functions called Compute, Print, and FilterRows. They look like this:

        static Either<string, A> FilterRows<A>(Transform<A>.FilterRows op, InterpreterState state) =>
            Interpret(op.Next(state.Rows.Filter(r => r.Type == op.Value)), state);

        static Either<string, A> Print<A>(Transform<A>.Print op, InterpreterState state)
        {
            op.Messages.Iter(state.Output.OnNext);
            return Interpret(op.Next(unit), state);
        }

        static Either<string, A> Compute<A>(Transform<A>.Compute op, InterpreterState state) =>
            op.Operation.Operator == "Sum" ? SumCompute<A>(op, state)
          : op.Operation.Operator == "Expression" ? ExprCompute<A>(op, state)
          : Left($"Unknown operator: {op.Operation.Operator}");

        static Either<string, A> ExprCompute<A>(Transform<A>.Compute op, InterpreterState state) =>
            op.SourceType is ValueSourceType sourceType
                ? Interpret(op.Next(unit), state.SetVar(op.Operation.Id, sourceType.Value))
                : Left("Invalid source type for Compute");

        static Either<string, A> SumCompute<A>(Transform<A>.Compute op, InterpreterState state) =>
            op.SourceType is RowsSourceType sourceType
                ? Interpret(op.Next(unit), state.SetVar(op.Operation.Id, sourceType.Rows.Map(x => x.Amount).Sum()))
                : Left("Invalid source type for Compute");

Note how ExprCompute and SumCompute both call SetVar to save the computed result. ExprCompute doesn’t actually need to compute anything because it’s already been done by the compiled Transform<A> and SumCompute doesn’t need to do any filtering because it’s already done as part of the FilterRows setup in the Transform factory functions.

And, that’s it, you can now run it using:

        static void Main(string[] args)
        {
            var transform = Data.Load(Data.OperationsToPerform);

            var state = InterpreterState.Empty.With(Rows: Data.Rows);

            state.Output.Subscribe(Console.WriteLine);

            var result = Interpreter.Interpret(transform, state);
        }

It outputs:

Total Assets : 160000
Total Expenses : 40000
Balance (Total Assets - Total Expenses): 120000

[louthy]

@imranypatel Because I see this as a useful example for people learning this library I have decided to make it into an example project (and because it’s probably hard to follow when everything is in one gist). It currently resides in the Samples folder on the async-refactor branch

I have taken the idea a bit further than your initial requirements. It’s now possible to avoid the manual building of the operations followed by ‘loading’ them into the Transform. Now you can just use the scripts:

TotalAssets = sum(filter("Asset", rows));
TotalExpenses = sum(filter("Expense", rows));
Balance = TotalAssets - TotalExpenses;
log("Total Assets : {TotalAssets}");
log("Total Expenses : {TotalExpenses}");
log("Balance (Total Assets - Total Expenses): {Balance}");

The way I’ve achieved that is to support function invocation and variable assignment in the scripting system. The functions sum and filter are provided in a ScriptFunctions class that allows you to extend the set of available operations without much hassle:

    public static class ScriptFunctions
    {
        public static Seq<AccountingRow> filter(string type, Seq<AccountingRow> rows) =>
            rows.Filter(r => r.Type == type);

        public static int sum(Seq<AccountingRow> rows) =>
            rows.Map(r => r.Amount).Sum();

        public static int count(Seq<AccountingRow> rows) =>
            rows.Map(r => r.Amount).Count;

        public static int avg(Seq<AccountingRow> rows) =>
            sum(rows) / count(rows);
    }

Below is the test method for loading from a script:

        static void Test_Script()
        {
            var script = @" TotalAssets = sum(filter(""Asset"", rows));
                            TotalExpenses = sum(filter(""Expense"", rows));
                            Balance = TotalAssets - TotalExpenses;
                            log(""Total Assets : {TotalAssets}"");
                            log(""Total Expenses : {TotalExpenses}"");
                            log(""Balance (Total Assets - Total Expenses): {Balance}""); ";

            var result = from transform in Parse(script).Map(Compile<object>)
                         let state = InterpreterState.Empty.With(Rows: Data.Rows)
                         let disp = state.Output.Subscribe(Console.WriteLine)
                         from _ in Interpret(transform, state)
                         select unit; 
        }

Note, these functions and scripting extensions are available in the version where you manually build the operations. Although, you only really need the Print operation now:

Print("Print messages after substituting computed Ids in each message format string",
    Seq(
        "Total Assets : { sum(filter("Asset", rows)) }",
        "Total Expenses : { sum(filter("Expense", rows)) }",
        "Balance (Total Assets - Total Expenses): { sum(filter("Asset", rows)) - sum(filter("Expense", rows)) }"))