Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

IndexError: lark.parsers.earley_forest in visit_token_node

See original GitHub issue

Hi! I’m using earley parser in uwsgi app. And everything is fine, but periodically have the exceptions:

IndexError: deque index out of range
  File "app/helpers/predicates.py", line 16, in check_predicate
    return evaluate_predicate(predicate, attributes=attributes)
  File "predicates/evaluator.py", line 13, in evaluate_predicate
    ast = parser.parse(predicate)
  File "lark/lark.py", line 306, in parse
    return self.parser.parse(text, start=start)
  File "lark/parser_frontends.py", line 185, in parse
    return self._parse(text, start)
  File "lark/parser_frontends.py", line 54, in _parse
    return self.parser.parse(input, start, *args)
  File "lark/parsers/earley.py", line 319, in parse
    return self.forest_tree_visitor.visit(solutions[0])
  File "lark/parsers/earley_forest.py", line 281, in visit
    return super(ForestToTreeVisitor, self).visit(root)
  File "lark/parsers/earley_forest.py", line 204, in visit
    vtn(current)
  File "lark/parsers/earley_forest.py", line 284, in visit_token_node
    self.output_stack[-1].append(node)

I cannot reproduce the error manually.

My grammar:

predicate: _statement

_expr: attribute
    | bool_constant
    | attribute_comparison
    | version_attribute_comparison
    | time_comparison
    | functions
    | _composed_statement
    | not_statement

attribute.1: ATTRIBUTE_NAME

bool_constant.2: BOOLEAN

attribute_comparison.3: ATTRIBUTE_NAME (EQ|N_EQ) BOOLEAN                                      -> bools_comparison
                      | ATTRIBUTE_NAME (EQ|N_EQ) SIGNED_INT                                   -> nums_eq_neq_comparison
                      | ATTRIBUTE_NAME (GE|LE) SIGNED_INT                                     -> nums_ge_le_comparison
                      | ATTRIBUTE_NAME (EQ|N_EQ) _QUOTATION TEXT_WITHOUT_QUOTATION _QUOTATION -> strings_comparison

version_attribute_comparison.4: VERSION_ATTRIBUTE_NAME (EQ|N_EQ|GE|LE) VERSION -> versions_comparison

time_comparison.5: DATE_ATTRIBUTE_NAME (EQ|N_EQ|GE|LE) DATE         -> dates_comparison
                 | DATETIME_ATTRIBUTE_NAME (EQ|N_EQ|GE|LE) DATETIME -> datetimes_comparison

functions.6: _DEFINED _BRACE_OPEN ATTRIBUTE_NAME _BRACE_CLOSE   -> defined
           | _UNDEFINED _BRACE_OPEN ATTRIBUTE_NAME _BRACE_CLOSE -> undefined
                 
_composed_statement.7: _BRACE_OPEN _statement _BRACE_CLOSE

not_statement.8: _NOT _composed_statement

?and_statement.9: _expr [_AND and_statement]

?or_statement.10: and_statement [_OR or_statement]

_statement: or_statement

ATTRIBUTE_NAME: /(?!NOT\_)(\w(\w|[-.])*)([a-zA-Z]\d*)/
VERSION_ATTRIBUTE_NAME: "source_client_version"
DATE_ATTRIBUTE_NAME: /[a-zA-Z0-9_.]+_date/
DATETIME_ATTRIBUTE_NAME: /[a-zA-Z0-9_.]+_time/

_DEFINED: "defined"
_UNDEFINED: "undefined"

BOOLEAN: "true"|"false"
TEXT_WITHOUT_QUOTATION: /[^']+/
_QUOTATION: "'"
VERSION: INT "." INT ("." INT)~0..1
DATE: INT ("-" INT)~2
DATETIME: INT ("-" INT)~2 "T" INT (":" INT)~2

EQ: "="
N_EQ: "!="
GE: ">="
LE: "<="

_BRACE_OPEN: "("
_BRACE_CLOSE: ")"

_NOT: "NOT"

_OR: /\s+OR\s+/
_AND: /\s+AND\s+/

SIGNED_INT: ["+"|"-"] INT

%import common.INT

Parser initialized with the next params: parser = Lark(grammar, start="predicate", parser="earley")