Inspect parse tree as it's being parsed, without transforming.

I’m trying to use the transformer argument of the Lark class to have a visitor inspect everything that has been parsed so far, so that later this can help with the lexer (something similar to “the lexer hack” some C compilers use). I tried using an InPlace trasnformer and not-transforming anything, but it seems that the InPlace transformer is not accepted, or rather it is accepted but used as a base transformer. This is a very simple example:

grammar = r"""
program: idx+
idx: "xyz"i
%ignore /[ \t\f]+/
"""
prog="xyz xyz xyz"

@v_args(tree=True)
class T(Transformer_InPlace):
    def idx(self, t):
        print('visited:', t)

parser = Lark(grammar, start='program', parser='lalr', transformer=T())
t = parser.parse(prog)
print(t.pretty())

This will output:

visited: []
visited: []
visited: []
program
  []
  []
  []

This means that neither v_args has worked as it should, nor has Transformer_InPlace behaved as expected.

Is this the expected behavior? Because if it is, I don’t think it’s documented anywhere. Also, is there any other way of achieving this?

[erezsh]

although it’s a bit unintuitive

I guess what you’re trying to do is a bit unusual, so it’s not that bad. If someone comes up with a better design I’ll be happy to consider it. But as for the current one, as they say, I had my reasons 😃

As for questions, no worries. I always have time for contributors, and I’ll be happy to help.

[elektito]

I guess you’re right, although it’s a bit unintuitive. Anyways, I already fixed my code with the reconstruction code you first suggested and moved on. I believe I will be back with other questions soon, so I apologize for that beforehand! 😃