Lark cache fails when custom lexers are used
See original GitHub issueDescribe the bug
In parser_frontends.py, the passed lexer is wrapped using functools.partial:
return partial(LALR_CustomLexer, lexer)
but this results in a functools.partial object, not a lexer object, so when deserialize is called later, it fails with
AttributeError: 'functools.partial' object has no attribute 'deserialize'
--> 338 self.parser = self.parser_class.deserialize(data['parser'], memo, self._callbacks, self.options.postlex, self.re)
To Reproduce I’ve modified the custom lexer example
from lark import Lark, Transformer, v_args
from lark.lexer import Lexer, Token
class TypeLexer(Lexer):
def __init__(self, lexer_conf, re_=None):
pass
def lex(self, data):
for obj in data:
if isinstance(obj, int):
yield Token('INT', obj)
elif isinstance(obj, (type(''), type(u''))):
yield Token('STR', obj)
else:
raise TypeError(obj)
grammar = """
start: data_item+
data_item: STR INT*
%declare STR INT
"""
parser = Lark(grammar, parser='lalr', lexer=TypeLexer, cache=True)
# Fails
parser = Lark(grammar, parser='lalr', lexer=TypeLexer, cache=True)
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (4 by maintainers)
Top Results From Across the Web
Developers - Lark cache fails when custom lexers are used -
Describe the bug. In parser_frontends.py, the passed lexer is wrapped using functools.partial: return partial(LALR_CustomLexer, lexer).
Read more >Lark Documentation
Flexible error handling by using an interactive parser interface (LALR ... lexer_callbacks can be used to interface with the lexer as it ...
Read more >lark-parser/Lobby - Gitter
I approached the problem like xml xpath and a document object model. I used that example, started at the root, tried to walk...
Read more >Ubuntu Manpage: lark - Lark Documentation
This is a huge improvement to Earley that is unique to Lark. This feature is used by default, but can also be requested...
Read more >lark.js - Documentation
debug (bool): in case of error, should the parser output debug info to the console? ... to be used by the parser */...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
~~
functools.partial
is not picklable, according to python’s pickling doc: https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled~~You have to avoid using
functools.partial
.Instead, you can define a function at the top level of a module:or subclass the
LALR_CustomLexer
:@adefazio It looks good to me, make a PR with a test.