question: is incremental parsing possible?

I have the use case where I need to parse a text line by line. At the first line, the parser can start at its start position. At the end of the line, the parser should store its state (e.g. in which context is is) in a hashable object. When parsing any line that is not the first line, the state of the end of the previous line is restored and the current line is parsed.

(This is how QSyntaxHighlighter (from Qt) is organized. When the user types in a certain line, that line is re-highlighted. And when its state at the end is different than it was beforehand, the next line is also re-highlighted, and so on.)

Would this be possible using the lark parser? From my own experimentation and docs reading I can only parse a full text (or file) in one go.

[MegaIng]

The existing interactive parser can be used to create a small wrapper that does this.

from queue import Queue

from lark import Discard, Lark

json_grammar = r"""
    ?start: "[" [command ("," command)*] "]"
    command: value

    ?value: object
          | array
          | string
          | SIGNED_NUMBER      -> number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

    array  : "[" [value ("," value)*] "]"
    object : "{" [pair ("," pair)*] "}"
    pair   : string ":" value

    string : ESCAPED_STRING

    %import common.ESCAPED_STRING
    %import common.SIGNED_NUMBER
    %import common.WS

    %ignore WS
"""


class Transformer:
    def __init__(self, callback):
        self.callback = callback

    def command(self, children):
        self.callback(children[0])
        return Discard


def iter_parser(*args, transformer, **kwargs):
    queue = Queue()
    if not kwargs.setdefault("parser", "lalr") == "lalr":
        raise ValueError("The lalr parser is required")
    kwargs['transformer'] = transformer(queue.put)
    parser = Lark(*args, **kwargs)

    def parse(text, start=None):
        interactive = parser.parse_interactive(text, start)
        token = None
        for token in interactive.iter_parse():
            while not queue.empty():
                yield queue.get()
        interactive.feed_eof(token)
        while not queue.empty():
            yield queue.get()
    return parse


p = iter_parser(json_grammar, parser="lalr", transformer=Transformer)

test_text = """
[
 {"command": "print", "args": ["argument", 0, {"some": "object"}]},
 {"command": "input", "args": ["some prompt"]}
]
"""

for c in p(test_text):
    print("got", c)

[jspaezp]

Im not sure if I am more impressed by the response time or the actual solution … This is amazing! I definitely feel like this could be part of the tutorials. Right now I do not have time to write it myself and submit a PR, but if there is interest I can give it a go at a later time point.