Prepare Emitter API to go public

In preparation for #3619 and #3620 I’d live to review out Emitter API and rename some things to better match our own internal naming. I think the original was a bit too specific with regards to internal implementations and the new version should be a little more abstract:

Old:

• addKeyword(text, scope)
• addText(text)
• addSublanguage(emitter, subLanguageName)
• finalize()
• openNode(scope)
• closeNode()
• closeAllNodes()
• toHTML()

New:

• addText(text)
• startScope(scope) (or pushScope?)
• endScope() (or popScope? or finishScope?)
• __addSublanguage(emitter, subLanguageName) - (may stay private)
• finalize() - thoughts on naming?
• toHTML()

Removed:

• <del>addKeyword(text, scope) </del> - pull back into an internal helper
• <del>closeAllNodes()</del> - Lets just let finalize be responsible for all cleanup.

For writing custom grammar parsers one would really only need to concern themselves with 3 API calls I think:

• addText
• startScope
• endScope

@wooorm I know you had plenty of thoughts on this before… it’s possible I’m a little more receptive now… I do plan on reducing a API a bit if we’re going public (which I recall you pushed for previously).

[joshgoebel]

as long as there are ways to bypass the default tokenizer and there is a way to provide the actual token

That’s what #3620 allows. It should land in 11.7 (as a beta feature) with a full release with v12.

[foo123]

@joshgoebel as long as there are ways to bypass the default tokenizer and there is a way to provide the actual token (parsed with an external parser), my addon is fine. I don’t have the time at this point to test it, but I will do it once it is live.

I wonder how BNF parsers tend to do with ‘getting back on track’ when presented with wholly invalid data? That’s often a thing with code snippets - incomplete context, typos, etc…

This is part of error recovery. The approach can be simple or complex. A simple approach is discard everything (eg highlight as default text) until a valid sequence is found, then start again.