Newbie questions

Consider the below snippet:

from lark import Lark, inline_args, Transformer

grammars = [
    """
        ?start: sum | NAME "=" sum
        ?sum: product | sum "+" product | sum "-" product
        ?product: atom | product "*" atom | product "/" atom
        ?atom: NUMBER | "-" atom | NAME | "(" sum ")"
        %import common.CNAME -> NAME
        %import common.NUMBER
        %import common.WS_INLINE
        %ignore WS_INLINE
    """,
    """
        ?start: sum | NAME "=" sum
        ?sum: product | sum "+" product | sum "-" product
        ?product: atom | product "*" atom | product "/" atom
        ?atom: NUMBER | "-" atom | NAME | "(" sum ")"
        EQUAL: "="
        LPAR: "("
        RPAR: ")"
        SLASH: "/"
        STAR: "*"
        MINUS: "-"
        PLUS: "+"
        %import common.CNAME -> NAME
        %import common.NUMBER
        %import common.WS_INLINE
        %ignore WS_INLINE
    """,
    """
        ?start: sum | NAME "=" sum
        ?sum: product | sum "+" product | sum "-" product
        ?product: atom | product "*" atom | product "/" atom
        ?atom: NUMBER | "-" atom | NAME | "(" sum ")"
        OPERATOR : "=" | "(" | ")" | "/" | "*" | "-" | "+"
        %import common.CNAME -> NAME
        %import common.NUMBER
        %import common.WS_INLINE
        %ignore WS_INLINE
    """
]


def test(grammar, text):
    parser = Lark(grammar, start='start')
    # print(parser.parse(text).pretty())
    print(sorted(list(set([t.type for t in parser.lex(text)]))))
    # print([t.name for t in parser.lexer.tokens])


text = "x = 1+2 - 3-4 - 5*6 - 7/8 - (9+10-11*12/13)"
for i, grammar in enumerate(grammars):
    print('grammar {}'.format(i).center(80, '*'))
    test(grammar, text)

whose output is:

***********************************grammar 0************************************
['NAME', 'NUMBER', '__EQUAL', '__LPAR', '__MINUS', '__PLUS', '__RPAR', '__SLASH', '__STAR']
***********************************grammar 1************************************
['EQUAL', 'LPAR', 'MINUS', 'NAME', 'NUMBER', 'PLUS', 'RPAR', 'SLASH', 'STAR']
***********************************grammar 2************************************
['NAME', 'NUMBER', '__EQUAL', '__LPAR', '__MINUS', '__PLUS', '__RPAR', '__SLASH', '__STAR']

got some questions:

1. About grammar0, this set of token types '__EQUAL', '__LPAR', '__MINUS', '__PLUS', '__RPAR', '__SLASH', '__STAR' are generated automagically, how does this work internally?

2. About grammar1, following this method I’ll be able to identify easily the token types so I can use the types to syntax highlight with QScintilla, is there any problem with this approach?

3. About grammar2, in case I want to syntax highlight a group of similar tokens, how can I do that? In this case the token types are still generated automatically instead becoming OPERATOR. I’d like to be able to apply one QScintilla style to a bunch of related tokens (ie: OPERATORS= " | “(” | “)” | “/” | “*” | “-” | “+”)

[erezsh]

Lack of time or interest to answer?

Yes, somewhat.

I’ve been force to learn a little bit more about this passionate world of parse&compilers

Also part of the reason.

I think in the last month, you’ve asked more questions than all the other users combined. Also, many of the questions were just about Python or coding in general, and not Lark. And many of the questions about Lark already had the answer in the docs and tutorials. I don’t mind answering, but it takes time and effort than I can better spend on other things, such as improving Lark. Please try to do your “homework” and spend some time on solving the problem before asking here. And when you ask, try to focus on the specific line or concept that are the issue, instead of posting a 200 line code paste and asking “what isn’t it working?” I hope you won’t take offense, and that you keep asking questions and participating in Lark, under these guidelines.

SO is a really good way to increase the traffic so more devs will give it a shot.

Yes, but limited time, etc. If you find a question there that seems very relevant, feel free to ping me and I’ll check it out.

[erezsh]

Good to hear. I enjoy answering questions when they are well-researched or already reduced to the core of the problem. Yours were some yes, some no, and I was a little overwhelmed. I’m glad you took it positively.

Feel free to review, criticize, and make suggestions to Lark. That’s always good to have.

Regarding a #lark channel, no problem. I was going to add it, I just wanted to provide a link to a web client along with it, and didn’t have time to do the “research” for which is the best one. But I have to say, I think the channel will be empty. Lark isn’t nearly as popular (or widely applicable) as numpy or even pandas.

my English is creepy.

Your English is pretty good, but I think you meant crappy, not creepy 😆