Recovery Re-Sync logic ignores Token Categories

It seems the tokenMatcher function is not used which means the token categories are not handled properly.

Example:

• https://github.com/SAP/chevrotain/blob/c001c1a8f94f0d00de03a10be701bcb28a5e34be/packages/chevrotain/src/parse/parser/traits/recoverable.ts#L278-L291

Possibly also affects this:

• https://github.com/SAP/chevrotain/blob/c001c1a8f94f0d00de03a10be701bcb28a5e34be/packages/chevrotain/src/parse/parser/traits/recoverable.ts#L269-L276

This may cause the Parser to re-sync (throw out) more input than needed to recover.

[AlansCodeLog]

Edit: Nevermind, I think the problem was something else and also maybe I misunderstood how the recovery is supposed to work*, but, I got things working again and recovering how I wanted. <strike> Just ran into this. I’m writing a parser that takes both symbol and non-symbol operators, so they are distinct token types (since the word operators must use longer_alt but not the symbol ones), but the operators have a category to be able to consume them since MANY_SEP won’t take rules, unfortunately using it leads to this issue.

The following workaround works but is very wordy since DEF must be written twice.

// $.MANY_SEP({ SEP: OPERATOR_AND, DEF })
$.OR([
	{
		GATE: () => $.LA(2).tokenType === SYM_AND,
		ALT: () => $.MANY_SEP({ SEP: SYM_AND, DEF }),
	},
	{
		GATE: () => $.LA(2).tokenType !== SYM_AND,
		ALT: () => $.MANY_SEP2({ SEP: WORD_AND, DEF2 })
	},
])

Is there any danger in just using js control flows like this if it seems to work?

if ($.LA(2).tokenType === SYM_AND) {
	$.MANY_SEP({ SEP: SYM_AND, DEF })
} else {
	$.MANY_SEP({ SEP: WORD_AND, DEF })
}

</strike>

[bd82]

I will see where I can contribute.

Awesome 👍