UnexpectedCharacters is misleading

I run into a weird situation. Lark is raising an UnexpectedCharacters error with this message:

lark.exceptions.UnexpectedCharacters: No terminal defined for '*' at line 3 col 3

That terminal is defined in my grammar, and if I change

3 * 3

to

3*3

it gets parsed correctly.

For reference, the grammar file being used is:

start: _NL? block+
line: values | operation | comment | assignment | imports | return_statement | block
number: INT | FLOAT
string: SINGLE_QUOTED | DOUBLE_QUOTED
boolean: TRUE | FALSE
list: _OSB (_NL _INDENT)? (values (_COMMA (_WS|_NL)? values)*)? (_NL _DEDENT)? _CSB
key_value: string _COLON values
objects: _OCB (key_value (_COMMA key_value)*)? _CCB
mutation: _WS NAME arguments*
values: (number | string | boolean | FILEPATH | list | objects) mutation?
comparisons: GREATER | GREATER_EQUAL | LESSER | LESSER_EQUAL | NOT | EQUAL
path_fragment: _DOT NAME | _OSB INT _CSB | _OSB string _CSB
path: NAME (path_fragment)*
assignment_fragment: EQUALS _WS? (values|path|service)
assignment: path _WS? assignment_fragment
return_statement: _RETURN _WS (path|values)
operator: PLUS | DASH | MULTIPLIER | BSLASH
operation: values _WS operator _WS values | values operator values
path_value: path | values
if_statement: _IF _WS path_value (_WS comparisons _WS path_value)?
nested_block: _INDENT block+ _DEDENT
elseif_statement: _ELSE _WS? _IF _WS path_value (_WS comparisons _WS path_value)?
elseif_block: elseif_statement _NL nested_block
else_statement: _ELSE
else_block: else_statement _NL nested_block
if_block: if_statement _NL nested_block elseif_block* else_block?
foreach_statement: _FOREACH _WS NAME output
foreach_block: foreach_statement _NL nested_block
typed_argument: NAME _COLON types
function_argument: _WS typed_argument
function_output: _WS _ARROW _WS types
function_statement: FUNCTION_TYPE _WS NAME function_argument* function_output?
function_block: function_statement _NL nested_block
inline_expression: _OP service _CP
arguments: _WS? NAME? _COLON (values|path|inline_expression)
command: _WS NAME
output: (_WS _AS _WS NAME (_COMMA _WS? NAME)*)
service_fragment: (command arguments*|arguments+) output?
service: path service_fragment
service_block: service _NL (nested_block)?
when_block: _WHEN _WS (path output|service) _NL nested_block
block: line _NL|if_block|foreach_block|function_block|arguments|service_block|when_block
imports: _IMPORT _WS string _WS _AS _WS NAME
types: INT_TYPE | FLOAT_TYPE | STRING_TYPE | LIST_TYPE | OBJECT_TYPE | REGEXP_TYPE | FUNCTION_TYPE
comment: COMMENT+

_WS: (" ")+
_NL: /(\r?\n[\t ]*)+/
_INDENT: "<INDENT>"
_DEDENT: "<DEDENT>"
INT.2: "0".."9"+
FLOAT.2: INT "." INT? | "." INT
SINGLE_QUOTED: /'([^']*)'/
DOUBLE_QUOTED: /"([^"]*)"/
TRUE: "true"
FALSE: "false"
FILEPATH: /`([^"]*)`/
_COMMA: ","
_OSB: "["
_CSB: "]"
_COLON: ":"
_OCB: "{"
_CCB: "}"
GREATER: ">"
GREATER_EQUAL: ">="
LESSER: "<"
LESSER_EQUAL: "<="
NOT: "!="
EQUAL: "=="
NAME.1: /[a-zA-Z-\/_0-9]+/
_DOT: "."
EQUALS: "="
_RETURN: "return"
PLUS: "+"
DASH: "-"
MULTIPLIER: "*"
BSLASH: "/"
_IF: "if"
_ELSE: "else"
_FOREACH: "foreach"
_AS: "as"
_ARROW.2: DASH GREATER
_OP: "("
_CP: ")"
_WHEN: "when"
_IMPORT: "import"
INT_TYPE: "int"
FLOAT_TYPE: "float"
NUMBER_TYPE: "number"
STRING_TYPE: "string"
LIST_TYPE: "list"
OBJECT_TYPE: "object"
REGEXP_TYPE: "regexp"
FUNCTION_TYPE: "function"
COMMENT: /(?<=###)\s(.*|\n)+(?=\s###)|#(.*)/

[evandrocoan]

This problem is because your start symbol finite_solid_object either accepts box or cylinder, but not both at the same time. Now, you need to define how your grammar should accept them. One quick solution is to use:

finite_solid_object : box
                    | cylinder
-->
finite_solid_object : ( box
                    | cylinder )+

Or:

finite_solid_object : finite_solid_object box
                    | finite_solid_object cylinder
                    | box
                    | cylinder

[evandrocoan]

Actually, you could do this to show everything in a single exception:

raise UnexpectedToken(t, e.allowed)
# -->
first_exception = "".join( traceback.format_exception( *sys.exc_info() ) )
raise UnexpectedToken( t, e.allowed, first_exception ) from None 

On this case, the constructor of UnexpectedToken would have to be changed to handle the new parameter.

The problem is that raise ... from None was added after Python 3.3:

Changed in version 3.3: None is now permitted as Y in raise X from Y

1. https://docs.python.org/3/reference/simple_stmts.html#the-raise-statement
2. https://stackoverflow.com/questions/24752395/python-raise-from-usage/24752607#24752607