Allow lexer to retokenize tokens with diff of text

Problem: When using chevrotain in an IDE/editor, calling Lexer.tokenize will throw out the previous lexer result whenever the text is updated, even if only a small portion of the text is updated.

Proposal: Add a new method to Lexer called retokenize or retokenizeRange, perhaps with a signature that looks like this retokenizeRange(prevResult: ILexingResult, range: { start: number, end: number, replacement: string }, mode: string), which takes the previous result and updates it according to the diff. The Ohm parser implements this API with m.replaceInputRange(startIdx: number, endIdx: number, str: string): https://github.com/harc/ohm/blob/master/doc/api-reference.md

Related to https://github.com/SAP/chevrotain/issues/598.

[christianvoigt]

Hi, I just wanted to share my experience so far: I am using Chevrotain for linting Markdown-like documents in a VS Code extension using a language server (https://marketplace.visualstudio.com/items?itemName=christianvoigt.argdown-vscode). Currently the language server will always validate the complete document.

The extension has already been used in philosophy seminars for analyzing argumentation. Texts of philosophy students can become quite long, but there haven’t been any problems yet. In the next version there will also be a live preview of svg graphics generated based on the code. This is also working by parsing the whole document on each change (though it is throttled). At least for my syntax, I currently don’t see the need to add partial re-parsing or re-lexing for any of these use cases.

[bd82]

I suggest trying to implement this on a specific grammar and use case and only afterwards attempting to generalize it for any grammar (if possible).

Lets re-open this issue if and when

1. You will encounter performance issues in an IDE scenario.
2. You have implemented some kind of partial parsing similar to the description here.

• I can help you with this if your code is open source or at least accessible.