Help with Transformer for simple query DSL?

Hi there!

I am a total newb when it comes to language-craft, and was wondering if I could get a little guidance on what I’m trying to do. I’ll give you the high level of my proof-of-concept, followed by my current test code - hopefully this will facilitate targeted advice.

High Level

I want to be able take a boolean expression like so: foo AND NOT (bar or baz >=6)

And convert it to a Python function like so:

def tester(test_obj):
    return test_obj["foo"] and not (test_obj["bar"] or test_obj["baz"] >= 6)

This pretty much means that I need to establish a little query DSL that combines boolean expressions and comparators, from what I understand. I have the tail ends of this process pretty much locked down: the grammar I’ve produced seems to be what I want, and compiling/exec-ing the function body should be very straightforward.

Here’s my question: How do I create a transformer that takes my boolean expression tree and turns it into the string body of a python function?

(If there’s another approach or construct I should be using, I’m all ears!)

What I have so far

A Simple filter grammar: filter_grammar.lark

?start: boolean_expression

?boolean_expression: term [OR_OP term]
?term: factor [AND_OP factor]
?factor: constant 
    | NOT_OP factor 
    | "(" boolean_expression  ")" 
    | NAME
    | comparison
?constant: FALSE | TRUE | NUMBER

?comparison: factor comparator factor

?comparator: EQUALS | LT | LTE | GT | GTE

// Custom Terminals
//

FALSE: "False" | "false"
TRUE:  "True"  | "true"
AND_OP: "AND" | "&"
OR_OP: "OR" | "|"
NOT_OP: "NOT"

EQUALS: "="
LT: "<"
LTE: "<="
GT: ">"
GTE: ">="

// Common Terminals
//

%import common.NUMBER
%import common.CNAME -> NAME
%import common.WS
%ignore WS

A test file: test_parser.py

from lark import Lark, Transformer
import ast


def get_parser():
    with open("filter_grammar.lark") as f:
        grammar = f.read()

    parser = Lark(grammar)
    return parser


# TROUBLE AREA ######

#
# How do I build a transformer that converts a boolean
# algebra expression into a python function, e.g.,
# from something like this:
#
# foo AND NOT (bar or baz >=6)
#
# to something like:
#
# def tester(test_obj):
#     return test_obj["foo"] and not (test_obj["bar"] or test_obj["baz"] >= 6)
#


class FunctionTransformer(Transformer):
    def boolean_expression(self, terms):
        print(terms)
        return terms

    def term(self, factors):
        return factors

    def factor(self, things):
        return things

    def name(self, string):
        return f"child_obj['{string}']"

    def number(self, number):
        return int(number)

    def and_op(self, _):
        return "and"

    def or_op(self, _):
        return "or"

    def not_op(self, _):
        return "not"


###
# END TROUBLE AREA
###

boolean_algebra_expression = """
    ((deaf OR hearing_impaired) AND age_years <=3)
        OR
    (NOT speaks_en AND age_years = 4)
"""

parser = get_parser()

tree = parser.parse(boolean_algebra_expression)

print(tree.pretty())


# Create property tester function using transformed
# boolean algebra --> python function.

boolean_expression_body = FunctionTransformer().transform(tree)

print(type(boolean_expression_body))

function_parts = [
    "def property_tester(self, child_dict):  return",
    boolean_expression_body,
]


new_code = ast.parse(" ".join(function_parts), mode="exec")

new_func = compile(new_code, filename="temp.py", mode="exec")

namespace = {}

exec(new_func, namespace)

property_tester = namespace["property_tester"]

test_child_one = {
    "deaf": True,
    "hearing_impaired": False,
    "age_years": 2,
    "speaks_en": True,
}

assert property_tester(test_child_one)

test_child_two = {
    "deaf": False,
    "hearing_impaired": True,
    "age_years": 3,
    "speaks_en": True,
}

assert property_tester(test_child_two)

test_child_three = {**test_child_two, "age_years": 4}

assert not property_tester(test_child_three)
# Too old, should give False

test_child_four = {**test_child_two, "speaks_en": False}

assert property_tester(test_child_four)
# Doesn't speak english and is also 4 - should give True

In particular, I’m wondering: should I be creating a transformer rule for every nonterminal/head of production? How do I handle nested binary operations?

Thanks in advance!

• Rico

[Datamance]

OK so now I realize I was mistaken about the cookbook format which is in Problem, Solution, Discussion format (so it’s paragraphs - not as structured as the table I had there). Maybe closer to what I was thinking of is Django ORM for SQL users, namely in that it’s just a nice little side-by-side comparison per SQL feature.

I’m trying to think of where I saw an actual table and coming up empty…

[erezsh]

Good idea, I’ll write something up when I have the time.