Postprocessor function fires for every matched character

One of the things that’s unexpected for me as I’m testing is that, if the post processor is matching a sequence, it fires for every matched identifier, including partial matches. Is that expected? For example, if I’m matching the :extend, and I do {% d => console.log(d) %}, I get logs like (see the second item in the nested array):

It’s not an ambiguous grammar problem (I presume), because it produces one “parsing”. I was naively creating objects on each post-processor call, but is this behavior expected? Should I be looking for a “Ok are you really done?” flag? Is this for error handling? The brief documentation doesn’t give any indication whatsoever that partial matches of a Nearley rule will fire the post-processor, and I don’t want to create objects that are never used. Can you explain?

[kach]

Yeah, a lexer would prevent this because :extend would be read by nearley in one chunk.

Tim wrote very good docs for moo — I don’t think you’ll have much trouble if you choose to go that route. On the other hand, I’d caution you to not spend too much time optimizing something if it’s not actually too slow. 😃

[tjvr]

I was trying to build a parser that had side-effects and was curious to venture down this road. Specifically, it was for AST node caching

I think “no side effects” is a bit strong, actually; what we really mean is that your AST nodes—aka what’s returned by your postprocessors—should be immutable.

Nearley may “reuse” an AST node in another parse, so if you mutate your AST nodes in-place you’re going to have a bad time.

This doesn’t necessarily stop you from writing some sort of caching system that “shares” child nodes between parent nodes; indeed if Nearley returns multiple parses, it’s likely that they will share children.

However, if what you’re really trying to do is something like common subexpression elimination, I’d probably do that as a separate step after parsing.

Sorry if the above doesn’t make sense; this stuff is hard to explain!

_{Sent with GitHawk}