Change in error handling between 0.4.18-153 and 0.4.18-154

It looks like there was a change in error handling between 0.4.18-153 and 0.4.18-154. I’m not sure if I’m doing something wrong, or if this is unexpected and I should try to reduce this.

I use Jison in linter-csound. On macOS, if I enter in Terminal

git clone https://github.com/nwhetsell/linter-csound.git
cd linter-csound/lib/csound-parser
npm install https://github.com/GerHobbelt/jison/archive/0.4.18-153.tar.gz
../../node_modules/jison-lex/cli.js preprocessor.jison-lex --outfile preprocessor.js
../../node_modules/jison/lib/cli.js orchestra.jison orchestra.jison-lex --outfile orchestra-parser.js
npm install csound-api
npm --global install jasmine
jasmine

to generate parsers and run some tests with Jison 0.4.18-153, all the tests pass. (If you want to run this, note that the csound-api package requires Boost and Csound. Here are installation instructions.)

If I then run

npm uninstall jison
npm install https://github.com/GerHobbelt/jison/archive/0.4.18-154.tar.gz
../../node_modules/jison-lex/cli.js preprocessor.jison-lex --outfile preprocessor.js
../../node_modules/jison/lib/cli.js orchestra.jison orchestra.jison-lex --outfile orchestra-parser.js
jasmine

to generate parsers and run some tests with Jison 0.4.18-154, I get one failure.

The failure appears to have something to do with using this grammar rule—

then_statement
  : THEN NEWLINE statements
    {
      $$ = new Then(@$, {children: $3});
    }
  | THEN NEWLINE
    {
      $$ = new Then(@$);
    }
  | THEN error
    {
      $$ = new Then(@$);
      parser.messages.push({
        type: 'Error',
        text: 'Expected newline',
        range: parser.lexer.rangeFromPosition(@1.last_line, @1.last_column)
      });
    }
  ;

—to parse this snippet:

      if 1 == 1 then + -
      endif

In Jison 0.4.18-154, it seems like an extra exception is thrown due to the error token in this grammar rule:

primary_expression
  : identifier
  | global_value_identifier
  | constant
  | '(' conditional_expression ')'
    {
      $$ = $2;
    }
  | error
    {
      parser.parseError('', {}, {
        type: 'Error',
        text: 'Expected expression',
        range: parser.lexer.rangeFromPosition(@$.first_line, @$.first_column)
      });
    }
  ;

Any ideas?

[GerHobbelt]

FWIW: as you have written tests for your grammar (👍 👍) make sure every error recovery rule in your grammar gets hit by at least one test case, while such test input should ideally at least travel through some (preferably all) other rules of your grammar which also have error recovery alternatives, which are or are not fired in the test case.

Your original input example is a fine test as it tests grammar behaviour post the first error recovery, not just whether initial error recovery occurs. (Such test cases are hard to construct intentionally and are often more complicated than the simplest unit tests. They are more like ‘system tests’, in a way.)

---

You may also want to check out yyerrok and yyclearin in the yacc/bison documentation when you go digging deeper into error recovery behaviour. My jison fork also supports those, but they’re pretty hairy to use, so I would suggest to develop any error recovery coding in small steps (and with lots of tests; I screw up almost daily myself as yacc error recovery coding is almost an art). 👿

There’s several error recovery test examples in the jison/examples/ directory, BTW.

Take care and good luck!

[nwhetsell]

Many, many thanks for your detailed response. I’m closing this, because I think the main issue at this point is that I have significant gaps in my understanding of how error recovery in Bison/Jison actually works.