How do I just tokenize, without using color themes?

Hi - shiki looks great! I’m trying to use it to replace prism.js on https://tigyog.app/.

But I’m a bit confused about how to just get the raw tokens, or to get HTML output that uses normal CSS variables. I need this so I can generate HTML that uses the existing CSS classes and theme.

I can see codeToHtml, but this adds raw CSS colors to the HTML depending on theme. And I can see codeToThemedTokens, but that also seems to give me colors, rather than syntactic token types like variable, comment etc. I eventually found the 'css-variables' theme, but this seems hacky – it gives me strings like "var(--shiki-token-comment)", where I just want the string "comment". I could parse the "comment" string out of this, but it feels like I’m using it wrongly.

Here’s the API I expected:

interface IToken {
    /**
     * The content of the token
     */
    content: string;
    /**
     * E.g. "comment", "variable", etc
     */
    type: string;

    explanation?: ITokenExplanation[];
}

codeToTokens(code: string, lang?: StringLiteralUnion<Lang>): IToken[][];

Basically, I just want to use the tokenizer functionality, without the theme features. Is there a way to do this idiomatically with the shiki API?

[Gerrit0]

If you don’t want any styling info, you probably don’t want Shiki… using vscode-textmate directly will give you the tokens you’re after, without attaching theming info. You’ll want grammar.tokenizeLine.

You could also set includeExplanation when calling codeToThemedTokens and check the explanation property, but performance will be worse, since that option essentially results in tokenizing everything twice, and slicing strings many times more than without that option

[wooorm]

Maybe my project https://github.com/wooorm/starry-night might be of help in this case. It’s similar but different. Exposes a AST. Uses the same dependencies.

Also wanted to bring this to the attention to maintainers here: some recent issues are solved by starry-night, so if features don’t really make sense in Shiki, you can send users to starry-night.