Lexer callbacks not called under Earley parser

The following code:

#! /usr/bin/env python3

from lark import Lark


GRAMMAR = """
    start:
    EKS: /x/
    %ignore EKS
"""

if __name__ == '__main__':
    Lark(GRAMMAR, parser='earley', lexer_callbacks={'EKS': print}).parse('x')

prints nothing, unlike its LALR equivalent:

#! /usr/bin/env python3

from lark import Lark


GRAMMAR = """
    start:
    EKS: /x/
    %ignore EKS
"""

if __name__ == '__main__':
    Lark(GRAMMAR, parser='lalr', lexer_callbacks={'EKS': print}).parse('x')

which prints the expected output, x.

It appears that under the Earley parser the lexer callbacks are not invoked, which in my particular case means that I lose the ability to collect comments when I switch to Earley.

[erezsh]

To the best of my knowledge, the Earley lexer currently doesn’t allow access to ignored tokens. The difficulty is that because they are not part of the tree, and there might be many possible trees (due to ambiguity), there is no way to know, at the time of lexing, which tokens are correct and which aren’t.

However, it might be possible to allow the comments to be saved on the tree itself, so they can be collected when the parse is over.

If I had to guess, I would say this would be the key point to patch: https://github.com/lark-parser/lark/blob/master/lark/parsers/xearley.py#L75

The main challenge would be to decide which tree node to attach the comments to, and to ensure that they are attached correctly.

[pileon]

@erezsh I have a similar problem to the OP, in that I want to be able to catch comments.

The problem is that the grammar doesn’t work with the lalr parser or the standard lexer.

Are there other ways to catch (and ignore in the parser!) terminals using the earley parser and its default lexer?