Add error recovery mode?

In my language, lines with invalid stuff are considered comments. I know, insane, but I’d like to support that if possible. Currently, { error: true } is extremely greedy and considers everything from the first error to be a single token.

I propose an optional error tolerance mode that can be enabled with { error: true, recover: true }:

• if can’t parse a token starting with current position position:
  • if not already in recovery mode, save current position as recovery starting point
  • increment position (skip current character)
• if can parse a token && recovery starting point exists:
  • return a token with name of error token type, line and col of recovery starting point
  • delete recovery starting point (so we return current valid token next time)

[tjvr]

Here are some suggestions:

1. We recommend not implementing error recovery as part of your lexer (as @nathan says).
2. Since keyword handling is now explicit (#57), your excerpt from above should now behave as expected.
3. If you really want to do this, I think the right place is another library on top of moo; not in the moo core itself. Sorry! 😃

[tjvr]

the whole line } // comment gets parsed as invalid which doesn’t match rCurly.

Argh. I knew making keyword handling implicit would be a bad idea. Perhaps this is another reason to make keyword handling explicit (#53).