Parser gives up instead of getting out of nested rule to try alternatives

Here’s a repo with this exact reproducible case: https://github.com/deltaidea/chevro-section-conditions

Syntax

A script consists of sections, which consist of commands. Sections cannot be nested. There’re also if statements, both top-level (with sections inside) and section-level (with commands inside). See a sample script below.

Here’s pseudo-code of how I see the syntax:

Script = TopLevelStatement+
TopLevelStatement = TopLevelIf | Section
TopLevelIf = 'if FOO' TopLevelStatement+ 'endif'

Section = '<SECTION>' SectionStatement+
SectionStatement = SectionIf | 'COMMAND'
SectionIf = 'if FOO' SectionStatement+ 'endif'

Example

<SECTION>
  COMMAND

  if FOO
    COMMAND
  endif

if FOO
  <SECTION>
    COMMAND

  <SECTION>
    COMMAND
endif

In the example above the Parser tries the following path (whitespace is ignored):

<SECTION>
  COMMAND

  if FOO
    COMMAND
  endif

  if FOO
    // Expecting 'COMMAND' here, Parser thinks we're still inside the section started on line 1.
    // Instead it gets '<SECTION>' here and gives up.

• How do I force the parser to rewind, pop the rule stack and try parsing the ‘if’ as a TopLevelStatement?
• I can adjust the syntax definition in any way, so if there’s a better way to parse the same language, please tell me!

[bd82]

These are actually really good questions.

Firstly a context free parser cannot take into account the rule hierarchy of nested repetitions. as in the example you showed, simply because it will no longer be context free as we will need to take into account the parent rule in the hierarchy which may change between different invocations.

Secondly it looks like you will have to scan more than LL(*) tokens ahead to decide between the alternatives even if they were both in the same alternation. because you can always add more “COMMANDS” tokens until you find the deciding “section”. This means that for any lookahead K, there is an input where K+1 tokens will be need to disambiguate the choices.

You may be able solve this by using backtracking. https://github.com/SAP/chevrotain/blob/6a8febdcc78351b2b8610debc23f6d1b9bb1ce4d/test/full_flow/backtracking/backtracking_parser.ts#L62

But I strongly suggest you try to avoid backtracking as it is both slow, error prone (reseting state) and mutually exclusive with automatic error recovery.

Suggested Approach.

Explicitly close the “Section” Just like you have IF and ENDIF, you can also have “END_SECTION”.

[deltaidea]

Glad to know I’m not the only one. 😄 Flat list it is then.

Nah, the indentation doesn’t mean anything to the game that is the canonical interpreter. Newlines are the only part that has meaning, and only as a separator. Like newline after return in JS - it only matters whether at least one is there.

My goal is to be able to parse all existing valid scripts to provide tooling to the whole scripting community. That’s why I can’t change the syntax, only the parser implementation.