question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Lark cache fails when custom lexers are used

See original GitHub issue

Describe the bug

In parser_frontends.py, the passed lexer is wrapped using functools.partial:

return partial(LALR_CustomLexer, lexer)

but this results in a functools.partial object, not a lexer object, so when deserialize is called later, it fails with

AttributeError: 'functools.partial' object has no attribute 'deserialize'
--> 338         self.parser = self.parser_class.deserialize(data['parser'], memo, self._callbacks, self.options.postlex, self.re)

To Reproduce I’ve modified the custom lexer example

from lark import Lark, Transformer, v_args
from lark.lexer import Lexer, Token

class TypeLexer(Lexer):
    def __init__(self, lexer_conf, re_=None):
        pass

    def lex(self, data):
        for obj in data:
            if isinstance(obj, int):
                yield Token('INT', obj)
            elif isinstance(obj, (type(''), type(u''))):
                yield Token('STR', obj)
            else:
                raise TypeError(obj)

grammar = """
        start: data_item+
        data_item: STR INT*
        %declare STR INT
        """
parser = Lark(grammar, parser='lalr', lexer=TypeLexer, cache=True)

# Fails
parser = Lark(grammar, parser='lalr', lexer=TypeLexer, cache=True)

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
pwwangcommented, Jul 8, 2020

~~functools.partial is not picklable, according to python’s pickling doc: https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled~~

You have to avoid using functools.partial. Instead, you can define a function at the top level of a module:

- # you may want to detail args and kwargs in your case
-def MyLexer(*args, **kwargs):
-  return LALR_CustomLexer(lexer, *args, **kwargs)

or subclass the LALR_CustomLexer:

class MyLexer(LALR_CustomLexer):
    def __init__(self, *args, **kwargs):
      super().__init__(lexer, *args, **kwargs)
0reactions
erezshcommented, Jul 8, 2020

@adefazio It looks good to me, make a PR with a test.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Developers - Lark cache fails when custom lexers are used -
Describe the bug. In parser_frontends.py, the passed lexer is wrapped using functools.partial: return partial(LALR_CustomLexer, lexer).
Read more >
Lark Documentation
Flexible error handling by using an interactive parser interface (LALR ... lexer_callbacks can be used to interface with the lexer as it ...
Read more >
lark-parser/Lobby - Gitter
I approached the problem like xml xpath and a document object model. I used that example, started at the root, tried to walk...
Read more >
Ubuntu Manpage: lark - Lark Documentation
This is a huge improvement to Earley that is unique to Lark. This feature is used by default, but can also be requested...
Read more >
lark.js - Documentation
debug (bool): in case of error, should the parser output debug info to the console? ... to be used by the parser */...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found