Understanding the parse result

As a test, I have a super-simple grammar that accepts numbers across lines.

However, the result has an incredible amount of array-nesting, which increases if the input has multiple lines.

For instance, the input “4” yields

[[[[{"type":"number","value":"4","text":"4","offset":0,"lineBreaks":0,"line":1,"col":1}]]],null]

Why four levels of arrays? And more if there are more lines.

In general, what are the rules for traversing the result as an AST? It’s hard to know how deep you need to start looking for actual objects. What does the arrays signify at each level?

Grammar below (yes, over-complicated for numbers, but this is the starting point for something else):

@{%
const moo = require("moo");

const lexer = moo.compile({
    ws: {match: /[ \t\n\r]+/,  lineBreaks: true },
    number: /[0-9]+/,
});
%}

@lexer lexer

start -> exprlist:* %ws:?
exprlist -> expr
    | exprlist %ws expr
expr -> %number

[kach]

Right. The first level of array-ness (which has null as element 2) matches the start nonterminal. The first element matches the exprlist:*. The second element, null, represents the fact that the %ws:? item was not matched — “4” has no whitespace after it.

The second level of array-ness comes from the exprlist:* item. :* as you know means “match zero or more of these”. That means the “result” is actually an array of results, where each element in the array represents one exprlist. Since the input “4” only matches one exprlist, that array has one element.

The third level of array-ness is from the fact that you matched one expr in an exprlist, from the rule exprlist -> expr. Why is this in an array? Well, even though there’s only one item in the rule (expr), nearley always returns an array as the result: one element for each item. So, the first element in that array represents a parsed expr. If the rule was exprlist -> expr "cow" then the array would have two elements: one for the expr and one for the string "cow".

You can probably guess the fourth level of array-ness already, now! It’s from the rule expr -> %number, which returns a single-element array for exactly the same reason that the rule exprlist -> expr does.

Okay, so, how do we fix this? The easy answer is to use postprocessors (see Tim’s link). For example,

expr -> %number {% function(d) {return d[0];} %}

will now make expr return the contents of the %number, not an array! (It extracts the first element of the result array by doing d[0]).

In fact, the function `function(d) {return d[0]; } %} is so common, nearley provides it automatically: you can simply write

expr -> %number {% id %}

[tjvr]

@Hardmath123 will be along later to explain better, but I recommend reading about postprocessors: 🙂

By default, nearley wraps everything matched by a rule into an array.