Strangeness with 'expected' value from exception

This a cut-down example from a parser intended to accept disk format definitions. I do not understand why it fails to report the ‘libdisk’ token in the expected list from the thrown exception. The parse tree certainly looks reasonable, yet it insists on calling out an ANON token. If there’s a difference in the way the grammar is formed for that one terminal I’m not seeing it.

#!/usr/bin/python3

from lark.lark import Lark
from lark import Transformer, UnexpectedToken, UnexpectedCharacters, UnexpectedEOF
import pprint

GRAMMAR = r"""
start: diskdef+

diskdef: "diskdef" defname pair+ "end"

defname: CNAME

pair: intparm intval -> intparm
    | strparm strval -> strparm
    | osparm OSVAL -> osparm

intparm: "seclen" -> seclen
       | "tracks" -> tracks

strparm: "libdsk:format" -> libdisk

osparm: "os" -> os

?intval: INT

?strval: CNAME

OSVAL: "2.2"|"3"|"isx"|"p2dos"|"zsys"

%import common.WS
%ignore WS

%import common.INT
%import common.CNAME
%import common.LETTER
%import common.DIGIT
"""
parser = Lark(GRAMMAR, parser='lalr')

def do_parse(data):
    try:
        j = parser.parse(data)
        print(j.pretty())
    except UnexpectedToken as u:
        print("Unxpected token at line: %d, column: %d\n" % (u.line, u.column))
        txt = u.get_context(data)
        print(txt)
        print("Expected one of:")
        for tok in u.expected:
            print(tok)
    except UnexpectedCharacters as u:
        print(u)
    except UnexpectedEOF as u:
        print(u)

good_data = r"""
diskdef pcw
  seclen 512
  tracks 40
  os 3
  libdsk:format pcw180
end
"""
do_parse(good_data)

bad_data = r"""
diskdef pcw
  foobar 512
  tracks 40
  os 3
  libdsk:format pcw180
end
"""
do_parse(bad_data)

Parse Tree:

start
  diskdef
    defname     pcw
    intparm
      seclen
      512
    intparm
      tracks
      40
    osparm
      os
      3
    strparm
      libdisk
      pcw180

Error report:

Unxpected token at line: 3, column: 3

  foobar 512
  ^

Expected one of:
OS
SECLEN
TRACKS
__ANON_0  <----- Why?

[snhirsch]

Are you getting a backslash when doing

print( parser.get_terminal(tok).pattern.value )

?

No - that works correctly.

To name the terminal, you need to define it. Like

rule: NAMED_TERMINAL
NAMED_TERMINAL: "some_value"

Thanks very much for the clarification! Really appreciate the work you’ve done on this package.

[erezsh]

Are you getting a backslash when doing

print( parser.get_terminal(tok).pattern.value )

?

To name the terminal, you need to define it. Like

rule: NAMED_TERMINAL
NAMED_TERMINAL: "some_value"