Discussion about PostLexer API

This is a greatly simplified version of what I want to do:

from lark import Lark, Token

class SimplePostLex:
    def process(self, stream):
        for t in stream:
            return Token.new_borrow_position('B', t.value, t)
    @property
    def always_accept(self):
        return 'A',
p = Lark("""
start: B
A: "A"
%declare B
""", parser='lalr', postlex=SimplePostLex())
p.parse('A')

This does not work since the lexer does not accept A-Tokens, even though I told him to do so. It works if I include an A Token somewhere in the grammar.

[MegaIng]

I don’t like always_accept as a feature of the PostLexer either. That is simply not the place it belongs. (It changes the behavior of the Lexer even though it is supposed to happen afterwards). It would be better inside the grammar.

I am not sure if I like the idea of being able to edit the ignore list:

• First of all, it doesn’t actually fix this problem.
• Secondly, it forces the ‘user’ to micromanage the lexer and introduces a lot of potentially bugs for the ‘user’ to watch out for. (adding unnecessary things to ignore, forgetting to remove them, adding them to early/late…)

[MegaIng]

Fixed in #708