How to generate the stylesheet for tokens when using both TextMate and Monarch grammars simultaneously

I’ve mostly added TextMate grammars support in an app of mine, but that app still uses a Monarch grammar too, and I haven’t been able to figure out how to generate the right CSS for the tokens properly, without having the tokens defined in the Monarch stylesheet override the ones in the TextMate stylesheet and viceversa.

I’m hoping I won’t have to rewrite the Monarch grammar to TextMate as that would have quite a few downsides:

• Most of the times that Monarch grammar will be the only one my users will need, so no need to load Oniguruma at all in that common scenario.
• The Monarch grammar is most probably going to be faster, as it uses JS regexes.
• Also importantly I already have the Monarch grammar at hand, but translating that for TextMate is going to be error prone at best and very time consuming.

I’m guessing the root issue is that I basically have 2 registries for grammars that don’t know about each other, but how could I merge them when the registry for TextMate only seems to handle TextMate grammars?

Any ideas on how to address this issue?

/cc @bolinfest, in case you might have already encountered this issue. Thanks for monaco-tm by the way!

[fabiospampinato]

It actually worked!

For posterity, I did the following:

• Shifted all counters by 100 for the TextMate stylesheet, this ensures the classes for tokens from both registries won’t overlap:

css.replace ( /mtk(\d+)/gi, ( _, nr ) => `mtk${Number ( nr ) + 100}` );

• Shifted the counters by 100 for tokens generated by TextMate grammars, this ensures both in-editor and off-editor tokens match the classes in the stylesheet:

for ( let i = 1, l = tokens.length; i < l; i += 2 ) {
  tokens[i] += 100 << 14;
}

• Padded by 100 the color map from the Monarch/global registry and concatenated that with the color map from the TextMate registry, this ensures the right colors are displayed in the Minimap:

const colorsMonarch = Monaco.TokenizationRegistry._colorMap.slice ( 0, 100 ),
      colorsTextMate = Style.getColors ( registry ),
      colorPadding = Monaco.Color.Format.CSS.parseHex ( '#00000000' ),
      colors = new Array ( 100 + colorsTextMate.length ).fill ( colorPadding );

for ( let i = 0, l = colorsMonarch.length; i < l; i++ ) {
  if ( !colorsMonarch[i] ) continue;
  colors[i] = colorsMonarch[i];
}

for ( let i = 0, l = colorsTextMate.length; i < l; i++ ) {
  if ( !colorsTextMate[i] ) continue;
  colors[100 + i] = colorsTextMate[i];
}

Monaco.TokenizationRegistry.setColorMap ( colors );

• What’s left basically is just reusing the TextMate theme for Monarch grammars, I haven’t addressed that yet but that should be a bit tedious at worst, and not really needed perhaps.

I love Monaco! ❤️

[fabiospampinato]

Actually maybe I could just increment the bits the encode the foreground color to achieve this. I’ll have to try this, it sounds promising.

https://github.com/microsoft/vscode/blob/2f077172cb0fcbd06f5820b0a5125215e8d38435/src/vs/editor/common/modes.ts#L172