TypeError in parse_tree_builder.py:54 when propagate_positions=True

lark version: 0.7.8

TL;DR My grammar supports parsing of text which may end without a trailing newline.

When I am calling parser this way:

# fast parser, just for tree building and checking syntax
parser = Lark.open(
    os.path.join(_self_dir, 'gdscript.lark'),
    postlex=Indenter(),
    parser='lalr',
    start='start'
)

everything is fine.

But when I add propagate_positions=True to that, I get such error:

    def test_function_name():
        code = """
    func SomeName():
        pass"""
>       outcome = lint_code(code)

test_linter.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.tox/py3/lib/python3.7/site-packages/gdtoolkit/linter.py:35: in lint_code
    parse_tree = parser_with_metadata_gathering.parse(gdscript_code)
../.tox/py3/lib/python3.7/site-packages/lark/lark.py:311: in parse
    return self.parser.parse(text, start=start)
../.tox/py3/lib/python3.7/site-packages/lark/parser_frontends.py:89: in parse
    return self._parse(token_stream, start, *[sps] if sps is not NotImplemented else [])
../.tox/py3/lib/python3.7/site-packages/lark/parser_frontends.py:54: in _parse
    return self.parser.parse(input, start, *args)
../.tox/py3/lib/python3.7/site-packages/lark/parsers/lalr_parser.py:36: in parse
    return self.parser.parse(*args)
../.tox/py3/lib/python3.7/site-packages/lark/parsers/lalr_parser.py:105: in parse
    reduce(arg)
../.tox/py3/lib/python3.7/site-packages/lark/parsers/lalr_parser.py:76: in reduce
    value = self.callbacks[rule](s)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <lark.parse_tree_builder.PropagatePositions object at 0x7fe137ca02d0>, children = [Token(_NL, '\n    '), Token(_INDENT, '    '), Tree(_func_stmt_chain, [Tree(pass_stmt, [])]), Token(_DEDENT, '')]

    def __call__(self, children):
        res = self.node_builder(children)
    
        if isinstance(res, Tree):
            for c in children:
                if isinstance(c, Tree) and c.children and not c.meta.empty:
                    res.meta.line = c.meta.line
                    res.meta.column = c.meta.column
                    res.meta.start_pos = c.meta.start_pos
                    res.meta.empty = False
                    break
                elif isinstance(c, Token):
                    res.meta.line = c.line
                    res.meta.column = c.column
                    res.meta.start_pos = c.pos_in_stream
                    res.meta.empty = False
                    break
    
            for c in reversed(children):
                if isinstance(c, Tree) and c.children and not c.meta.empty:
                    res.meta.end_line = c.meta.end_line
                    res.meta.end_column = c.meta.end_column
                    res.meta.end_pos = c.meta.end_pos
                    res.meta.empty = False
                    break
                elif isinstance(c, Token):
                    res.meta.end_line = c.end_line
                    res.meta.end_column = c.end_column
>                   res.meta.end_pos = c.pos_in_stream + len(c.value)
E                   TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

../.tox/py3/lib/python3.7/site-packages/lark/parse_tree_builder.py:54: TypeError

You can reproduce here: https://github.com/Scony/godot-gdscript-toolkit/tree/1361b5e5e950997aa4c71e1c91dfcfce2884ace5 by applying patch:

diff --git a/tests/test_linter.py b/tests/test_linter.py
index ecd6e36..52a48ef 100644
--- a/tests/test_linter.py
+++ b/tests/test_linter.py
@@ -8,8 +8,7 @@ def test_empty_code_linting():
 def test_function_name():
     code = """
 func SomeName():
-    pass
-"""
+    pass"""
     outcome = lint_code(code)
     assert outcome[0].name == 'function-name'
     assert outcome[0].line == 2

I suspect gathering of lines/columns fails when no trailing newline is available.

[MegaIng]

I currently can’t test it, but my guess would actually be that the problem is that the Indenter class from lark spits out DEDENT-Tokens at the end of the file which don’t have pos_in_stream set. This should be fixed by recording the end_pos of the last Token.

[Scony]

@MegaIng you’re right - I didn’t even read the error, just uncommented and seen red prints. My bad. Sorry for bothering you.