Discuss: Custom token callbacks, or mode value property?

Hello, i’m playing around with highlight.js (9.2.0) and want to integrate with a -grammar add-on (following previous work on syntax-highlighting, for example here) which enables to syntax-highlight code by defining a grammar specification for the language (e.g in BNF form).

i have already made some integration code (to be uploaded here), but so far, in order to use the grammar parser and integrate with hljs core highlighter some mode boilerplate code (for example multiple modes inside contains and dummy ;lexemesRe, beginRe, endRe functions). While it would be easier (and more flexible) if there was some property that allowed a callback or even a static value with the lexeme (i,e token) to be directly available from the mode itself (and passed directly to highlighter to be wrapped in <span>[token value]</span> for highlight).

To be more explicit, consider the fragment from hljs highlight method below:

// ..
    var mode_buffer = '';
    var relevance = 0;
    try {
      var match, count, index = 0;
      while (true) {
        top.terminators.lastIndex = index;
        // it seems the only way is to override mode.terminators.exec
       // in order to hook here with the token
       // maybe accepting an already parsed value/callback would make all this more flexibe
      // e.g if ( mode.value ) count = processLexeme(value.substr(index, match.index - index), mode.value());
      // or sth like that
        match = top.terminators.exec(value);
        if (!match)
          break;
        count = processLexeme(value.substr(index, match.index - index), match[0]);
        index = match.index + count;
      }
      processLexeme(value.substr(index));
      for(current = top; current.parent; current = current.parent) { // close dangling modes
        if (current.className) {
          result += '</span>';
        }
      }
      return {
        relevance: relevance,
        value: result,
        language: name,
        top: top
      };
    } catch (e) {

// ..

Nikos

[joshgoebel]

This issue is quite old and am not aware of such functionality. If this is newer and it helps I will give it a look. Can you explain how this works?

Yes, it’s very new. Read the plugin docs and check out the PR regarding callbacks for highlight itself. The callbacks for highlightBlock are already in master.

[joshgoebel]

See:

https://github.com/highlightjs/highlight.js/pull/2404 https://github.com/highlightjs/highlight.js/pull/2395

Soon it should be a LOT easier to do this than in the past… You’d use a before callback on highlight to insert your custom parser/tokenizer and render your parsed HTML. You’re still responsible for doing all the tokenizing and HTML rendering yourself.

There is still no way to tie DIRECTLY into the existing tokenizer real-time via a simple plugin, but if someone really needed to do that you can replace the whole token tree/html renderer now by swapping two lines of code in the source. The key lines being:

var emitter = new TokenTree();
// ...
result = new HTMLRenderer(emitter, options).value();

One could even imagine allowing to configure this:

configure({
  emitterClass: TokenTree,
  htmlRenderer:(emitter, options) => { new HTMLRenderer(emitter, options).value(); }
})

I’m not sure we want to do that (yet or ever), but I’m thinking about it. The API is nice (I think) but it exposes a lot of internals and would make it harder to change the internals in the future I think. I do think we’ll expose the parse tree emitter somehow (right now it’s exposed as emitter in the result)… so if someone wanted to play with the token tree afterwards, or replace the build-in HTML renderer, it’d be pretty easy to do that. We already have an issue for that.