Is it possible for the post-lexer to consume two tokens and then yield?

I want to have a post-lexer that’s essentially looks like this (with more bits of code inserted in between!)?

class PostLexer:
    def __init__(self):
        self.always_accept = ()

    def process(self, stream):
        for tok in stream:
            tok2 = next(stream)
            yield tok
            yield tok2

The way this code is, the parser doesn’t work correctly. I get “no terminal defined for…” errors where I do not if I swap the tok2 = next(stream) line with the yield tok line.

[erezsh]

I assume you’re using LALR.

LALR by default uses the contextual lexer, which depends on the state of the parser to tokenize. Changing the order means it tries to determine a terminal before the parser advanced to the next state.

If you have to do it this way, for whatever reason, you can try to use lexer="standard". It will revert to the traditional YACC/PLY lexer, which doesn’t care about the parser state. However, that means that you’re losing a bit of parsing power, and might experience more collisions.

[elektito]

Yes. I guess that’s what I’m going to do then. Thanks.