Big delay for display highlighting with SemanticTokensProvider

Reproducible in vscode.dev or in VS Code Desktop?

• Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

• Not reproducible in the monaco editor playground

Monaco Editor Playground Code

Expand Monaco Editor Playground Code

/** @type {monaco.languages.SemanticTokensLegend} */
const legend = {
	tokenTypes: [
		'comment',
		'string',
		'keyword',
		'number',
		'regexp',
		'operator',
		'namespace',
		'type',
		'struct',
		'class',
		'interface',
		'enum',
		'typeParameter',
		'function',
		'member',
		'macro',
		'variable',
		'parameter',
		'property',
		'label'
	],
	tokenModifiers: [
		'declaration',
		'documentation',
		'readonly',
		'static',
		'abstract',
		'deprecated',
		'modification',
		'async'
	]
};

/** @type {(type: string)=>number} */
function getType(type) {
	return legend.tokenTypes.indexOf(type);
}

/** @type {(modifier: string[]|string|null)=>number} */
function getModifier(modifiers) {
	if (typeof modifiers === 'string') {
		modifiers = [modifiers];
	}
	if (Array.isArray(modifiers)) {
		let nModifiers = 0;
		for (let modifier of modifiers) {
			const nModifier = legend.tokenModifiers.indexOf(modifier);
			if (nModifier > -1) {
				nModifiers |= (1 << nModifier) >>> 0;
			}
		}
		return nModifiers;
	} else {
		return 0;
	}
}

const tokenPattern = new RegExp('([a-zA-Z]+)((?:\\.[a-zA-Z]+)*)', 'g');

monaco.languages.registerDocumentSemanticTokensProvider('plaintext', {
	getLegend: function () {
		return legend;
	},
	provideDocumentSemanticTokens: function (model, lastResultId, token) {
		const lines = model.getLinesContent();
        console.time('highlight')
		/** @type {number[]} */
		const data = [];

		let prevLine = 0;
		let prevChar = 0;

		for (let i = 0; i < lines.length; i++) {
			const line = lines[i];

			for (let match = null; (match = tokenPattern.exec(line)); ) {
				// translate token and modifiers to number representations
				let type = getType(match[1]);
				if (type === -1) {
					continue;
				}
				let modifier = match[2].length ? getModifier(match[2].split('.').slice(1)) : 0;

				data.push(
					// translate line to deltaLine
					i - prevLine,
					// for the same line, translate start to deltaStart
					prevLine === i ? match.index - prevChar : match.index,
					match[0].length,
					type,
					modifier
				);

				prevLine = i;
				prevChar = match.index;
			}
		}

        console.timeEnd('highlight')
		return {
			data: new Uint32Array(data),
			resultId: null
		};
	},
	releaseDocumentSemanticTokens: function (resultId) {}
});

// add some missing tokens
monaco.editor.defineTheme('myCustomTheme', {
	base: 'vs',
	inherit: true,
	colors: {},
	rules: [
		{ token: 'comment', foreground: 'aaaaaa', fontStyle: 'italic' },
		{ token: 'keyword', foreground: 'ce63eb' },
		{ token: 'operator', foreground: '000000' },
		{ token: 'namespace', foreground: '66afce' },

		{ token: 'type', foreground: '1db010' },
		{ token: 'struct', foreground: '0000ff' },
		{ token: 'class', foreground: '0000ff', fontStyle: 'bold' },
		{ token: 'interface', foreground: '007700', fontStyle: 'bold' },
		{ token: 'enum', foreground: '0077ff', fontStyle: 'bold' },
		{ token: 'typeParameter', foreground: '1db010' },
		{ token: 'function', foreground: '94763a' },

		{ token: 'member', foreground: '94763a' },
		{ token: 'macro', foreground: '615a60' },
		{ token: 'variable', foreground: '3e5bbf' },
		{ token: 'parameter', foreground: '3e5bbf' },
		{ token: 'property', foreground: '3e5bbf' },
		{ token: 'label', foreground: '615a60' },

		{ token: 'type.static', fontStyle: 'bold' },
		{ token: 'class.static', foreground: 'ff0000', fontStyle: 'bold' }
	]
});

const editor = monaco.editor.create(document.getElementById('container'), {
	value: [
		'Available token types:',
		'    [comment] [string] [keyword] [number] [regexp] [operator] [namespace]',
		'    [type] [struct] [class] [interface] [enum] [typeParameter] [function]',
		'    [member] [macro] [variable] [parameter] [property] [label]',
		'',
		'Available token modifiers:',
		'    [type.declaration] [type.documentation] [type.member] [type.static]',
		'    [type.abstract] [type.deprecated] [type.modification] [type.async]',
		'',
		'Some examples:',
		'    [class.static.token]     [type.static.abstract]',
		'    [class.static.token]     [type.static]',
		'',
		'    [struct]',
		'',
		'    [function.private]',
		'',
		'An error case:',
		'    [notInLegend]'
	].join('\n'),
	language: 'plaintext',
	theme: 'myCustomTheme',
	// semantic tokens provider is disabled by default
	'semanticHighlighting.enabled': true
});

setInterval(() => {
    editor.getModel().setValue(`Available token types:
    [comment] [string] [keyword] [number] [regexp] [operator] [namespace]
    [type] [struct] [class] [interface] [enum] [typeParameter] [function]
    [member] [macro] [variable] [parameter] [property] [label]

Available token modifiers:
    [type.declaration] [type.documentation] [type.member] [type.static]
    [type.abstract] [type.deprecated] [type.modification] [type.async]

Some examples:
    [class.static.token]     [type.static.abstract]
    [class.static.token]     [type.static]

    [struct]

    [function.private]

An error case:
    [notInLegend]`)
}, 1000)

Actual Behavior

Way to reproduce it without code changing on monaco example semanticTokensProvider

https://user-images.githubusercontent.com/56319745/146181865-6d4e494e-8cf2-4ee4-9dcd-03689e8c252e.mov

Way with synthetic setValue https://user-images.githubusercontent.com/56319745/146181894-da3392c3-0c67-4e52-9ed0-59b9813e30f1.mov

Actually time of recalculating new tokens very small, but highlighting has a delay for displaying and that’s visible to the eye

Expected Behavior

MonarchTokensProvider has a delay noticeably less

https://user-images.githubusercontent.com/56319745/146182860-e150432e-ef4b-4d9c-81a6-6d9c97394f97.mov

As we see on video, we can’t hardly notice that the value has changed

Code:

Expand Monarch Code

// Register a new language
monaco.languages.register({ id: 'mySpecialLanguage' });

// Register a tokens provider for the language
monaco.languages.setMonarchTokensProvider('mySpecialLanguage', {
	tokenizer: {
		root: [
			[/\[error.*/, 'custom-error'],
			[/\[notice.*/, 'custom-notice'],
			[/\[info.*/, 'custom-info'],
			[/\[[a-zA-Z 0-9:]+\]/, 'custom-date']
		]
	}
});

// Define a new theme that contains only rules that match this language
monaco.editor.defineTheme('myCoolTheme', {
	base: 'vs',
	inherit: false,
	rules: [
		{ token: 'custom-info', foreground: '808080' },
		{ token: 'custom-error', foreground: 'ff0000', fontStyle: 'bold' },
		{ token: 'custom-notice', foreground: 'FFA500' },
		{ token: 'custom-date', foreground: '008800' }
	],
	colors: {
		'editor.foreground': '#000000'
	}
});

// Register a completion item provider for the new language
monaco.languages.registerCompletionItemProvider('mySpecialLanguage', {
	provideCompletionItems: () => {
		var suggestions = [
			{
				label: 'simpleText',
				kind: monaco.languages.CompletionItemKind.Text,
				insertText: 'simpleText'
			},
			{
				label: 'testing',
				kind: monaco.languages.CompletionItemKind.Keyword,
				insertText: 'testing(${1:condition})',
				insertTextRules: monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet
			},
			{
				label: 'ifelse',
				kind: monaco.languages.CompletionItemKind.Snippet,
				insertText: ['if (${1:condition}) {', '\t$0', '} else {', '\t', '}'].join('\n'),
				insertTextRules: monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet,
				documentation: 'If-Else Statement'
			}
		];
		return { suggestions: suggestions };
	}
});

const editor = monaco.editor.create(document.getElementById('container'), {
	theme: 'myCoolTheme',
	value: getCode(),
	language: 'mySpecialLanguage'
});

function getCode() {
	return [
		'[Sun Mar 7 16:02:00 2004] [notice] Apache/1.3.29 (Unix) configured -- resuming normal operations',
		'[Sun Mar 7 16:02:00 2004] [info] Server built: Feb 27 2004 13:56:37',
		'[Sun Mar 7 16:02:00 2004] [notice] Accept mutex: sysvsem (Default: sysvsem)',
		'[Sun Mar 7 16:05:49 2004] [info] [client xx.xx.xx.xx] (104)Connection',
		'[Sun Mar 7 16:45:56 2004] [info] [client xx.xx.xx.xx] (104)Connection ',
		'[Sun Mar 7 17:13:50 2004] [info] [client xx.xx.xx.xx] (104)Connection ',
		'[Sun Mar 7 17:21:44 2004] [info] [client xx.xx.xx.xx] (104)Connection ',
		'[Sun Mar 7 17:23:53 2004] statistics: Use of uninitialized ',
		"[Sun Mar 7 17:23:53 2004] statistics: Can't create file /home/httpd/twiki/data/Main/WebStatistics.txt - Permission denied",
        '<11>httpd[25859]: [error] [client xx.xx.xx.xx]'
	].join('\n');
}

setInterval(() => editor.getModel().setValue(getCode()), 1000)

Additional Context

By the way that’s noticeable when data coming from outside and we need to update editor value, not by user editing.

Is there any way to do something about this?

[alexdima]

@aaliakseyenka There’s no fundamental reason. It’s just a heuristic which we have in almost all async providers where non-explicit (i.e. automatic) provider requests are made 300ms or 500ms after typing stops, to avoid invoking the providers too often (i.e. with each keystroke). In this particular case, the constants are:

• 100ms for viewport semantic tokens providers.
• 300ms for document semantic tokens providers. We have had requests in the past to make these requests more reactive, for example in case a language server spends 5 seconds to compute document semantic tokens, then they’d want that we invoke them less often. This is the first time I heard that someone wants their async provider invoked more often.

I wonder why do you want this? Are you perhaps having a language where you only register a semantic tokens provider and don’t register a tokens provider ?

[aaliakseyenka]

@alexdima thanks for doc update - that’s helpful