Using lark for a language server

Hey, I’m wondering how conducive lark would be for creating a language server implementation for a given DSL.

For some background:

• I have a custom DSL I’m using to describe device interfaces and then using jinja2 to generate the specific interface into the target language
• The main implementation of this obviously in Python
• I’m using antlr4 to define the grammar, which is used to generate a python and typescript implementations.
• I have written a basic language server extension for vscode in typescript using the antlr completion engine antlr4-c3. This provides live validation and auto-completion which is really useful.

Problems:

• Antlr’s python implementation is quite slow (~350ms to parse + transform a sample file compared to ~50ms with lark)
• I’m having to duplicate a lot of work between python & typescript, though admittedly they are solving slightly different problems.

I would quite like to use to Lark and implement the language server as part of the main python package if possible.

I realize a language sever requires some specific parser features namely:

• Error tolerance: This can partly be resolved by making the grammar itself more error tolerant and solving those errors later. I notice lark also an error puppet which can maybe be used for this purpose.
• Support for auto completion information: Its is namely done by parsing an incomplete source file up to the current caret position and extracting both the next possible tokens and the currently parser rule context from the parser. This can be combined with info from the last successful parse to produce auto completion options.

I’m wondering if lark as any support for these features? And If so how I might go about implementing them?

If something like this is possible, it might also be cool to write a language server for the lark grammar itself to aid development.

Cheers, Rob

[omega16]

I have been thinking for a couple of weeks about write a lsp for lark. I know #331 is about that but it’s focus is on Lint.

This could be done as an example for lsp implementation with lark. I mean, all users of lark should know lark syntax to write his grammars. So it makes sense to use lark as an example.

Lark BNFE grammar is less complex than others languages, and still is not so simple, so it makes a great target to expose some tricks.

So we could use the lsp to solve #331 (and we could use #331 to begin).

We could change #76 from lark to lark lsp as configurable option and it makes sence to have such a thing as part of a lsp (I know, backtracking is hard, in fact it makes python regex a contex sensitive grammar, right?).

We could add it to piode online editor.

We could have go to definition and hover. To small grammar it dosen’t matter, but on big ones would be great!

We could even add some extra features like #559 so client editor could render selected rules on demand, save as file and open or insert in place (editor based). I just want to make my editor do this, so you can ignore it if you want (or propose more crazy! one like having the parsed tree as image an go to rule in editor on click at node, and yes, i want to do this too).

What did you all think?

[erezsh]

I went ahead and created a repo in the lark-parser organization: https://github.com/lark-parser/lark-language-server. I sent invites to everyone in this conversation.

Its aim is to provide language server services for all IDEs, for any Lark grammar (as much as possible).

It’s fine to add IDE-specific code to it, but it should be marked/located as such.

Since this is a joint effort, I think we should all work in our own branches, and merge to master only through pull-requests.