recoveryEnabled Issue

Simple example has unexpected results, either:

• Incorrect parsing
• A missing value for: recoveredNode

Tokens:

export const ID = createTok({ name: "ID", pattern: /[_a-zA-Z]\w+/ });
export const Assign = createTok({ name: "Assign", pattern: /:=/ });
export const String = createTok({ name: "String", pattern: /([xDQUV]?'([^'\\]*(?:\\.[^'\\]*)*)')/ });

Parser:

        $.RULE("program", () => {
            $.CONSUME(ID);
            $.CONSUME(Assign);
            $.CONSUME(String);
        });

Test String:

aaa bbb :=  'Hello';

When recoveryEnabled is false there is no parsed output (as expected). When recoveryEnabled is true it produces the following cst:

{
  "name": "program",
  "children": {
    "ID": [
      {
        "image": "aaa",
...
      }
    ],
    "Assign": [
      {
        "image": ":=",
...
      }
    ],
    "String": [
      {
        "image": "'Hello'",
...
      }
    ]
  },
  "recoveredNode": undefined
}

When I would have expected:

{
  "name": "program",
  "children": {
    "ID": [
      {
        "image": "bbb",
...
      }
    ],
    "Assign": [
      {
        "image": ":=",
...
      }
    ],
    "String": [
      {
        "image": "'Hello'",
...
      }
    ]
  },
  "recoveredNode": undefined
}

Or:

{
  "name": "program",
  "children": {
    "ID": [
      {
        "image": "aaa",
...
      }
    ],
    "Assign": [
      {
        "image": ":=",
...
      }
    ],
    "String": [
      {
        "image": "'Hello'",
...
      }
    ]
  },
  "recoveredNode": true
}

[bd82]

Hello @GordonSmith

I don’t see a problem with the CST that was outputted.

What is happening here is single token deletion as described here:

• https://sap.github.io/chevrotain/docs/tutorial/step4_fault_tolerance.html#single-token-deletion

This is why the bbb is removed, as the parsing and heuristic is mostly greedy.

• First aaa is consumed to match the ID
• Then bbb is encountred while trying to consume an Assign
• The next token is indeed an assign so the bbb is skipped.

The recoveredNode is related to re-sync error recovery see more docs on this here:

• https://sap.github.io/chevrotain/docs/tutorial/step4_fault_tolerance.html#general-re-sync

Note that you may need to modify how error recovery works in your specific grammar for optimal results, its just a set of greedy heuristics, it is not expected to be the best for all grammars, some examples:

• https://github.com/SAP/xml-tools/blob/master/packages/parser/lib/parser.js#L142-L156
• https://sap.github.io/chevrotain/documentation/6_5_0/classes/cstparser.html#cantokentypebeinsertedinrecovery

[bd82]

https://github.com/SAP/chevrotain/commit/09afb8d5187c596def3932625b53aac84e6d2d02