How to deal with invalid syntax due to end of stream?

This issue requires a bit of preamble, but please bear with me 🙂

Take this simple grammar:

grammar.ne

main -> grouping:+

grouping -> lparen text rparen

text -> [a-zA-Z]:+

lparen -> "("

rparen -> ")"

If we compile it and create a parser that uses it:

parser.js

const nearley = require("nearley");
const grammar = require("./grammar");

const parser = new nearley.Parser(nearley.Grammar.fromCompiled(grammar));

module.exports = parser;

And feed it some data, (e.g. using the node repl), it works as expected:

const parser = require("./parser")
parser.feed("(hi)"); // [[[[["("],[["h","i"]],[")"]]]]]

If we omit the opening parenthesis, thereby providing it with invalid syntax, it throws an error as expected:

const parser = require("./parser")
parser.feed("hi)");
/*
Error: invalid syntax at line 1 col 1:

  hi)
  ^
Unexpected "h"
*/

However, if we omit the closing parenthesis instead (parser.feed("(hi");), no error is thrown and parser.results remains an empty array.

I found a similar description in a GitHub issue (#253), which lead me to #261 in which the “Empty parser.results” entry was checked. So I searched the docs for “empty” and found this section https://nearley.js.org/docs/parser#catching-errors which states:

If there are no possible parsings given the current input, but in the future there might be results if you feed it more strings, then nearley will temporarily set the results array to the empty array, [].

This leads me to believe that what I described is happening because the parser is waiting for future input that might contain the closing parenthesis. The section also states:

If there are no possible parsings, and there is no way to “recover” by feeding more data, then nearley will throw an error whose offset property is the index of the offending token.

My question is this: How do I tell nearley that there will be no more input, so that it can wrap up its parsing?

If nearley knows there won’t be any more data, it would be great to see some sort of error instead of an empty array. Perhaps something along the lines of Error: invalid syntax ... expected ")" or Error: unexpected end of stream.

P.S.

While looking through the source code, I found a finish method, but it doesn’t seem to be documented and I don’t know enough about the project to figure out exactly what it does. Calling it just seems to return an empty array in this case. https://github.com/Hardmath123/nearley/blob/ee5afdb3c7546cb03f33464219f22190791b424f/lib/nearley.js#L368-L382

Thanks!

[tjvr]

So an empty array can only be caused by an unexpected end of input (or no input), right?

Assuming feed() didn’t throw an error, then yes.

Glad I could help! 😃

[tjvr]

That’s interesting, I thought finish() did throw an error when there are no results. I agree that’s how it should work!

@Hardmath123 What do you think about changing this (or deprecating finish, and adding a different method with this behaviour)?

We should document this better, too! (PRs welcome, just ask.)

@mickdekkers Thank you for your detailed description, and for reading the docs so throughly! At the moment, parsing works as follows:

• make a Parser instance from your Grammar instance
• feed() your source.
• Make sure to catch syntax errors
• Call finish(), or access the results property, to get the array of parses.
• Make sure there’s exactly one: zero means we couldn’t parse it (unexpected end of input); more than one means the input was ambiguous.

Hope that helps! 😃

_{Sent with GitHawk}