Can I get scope / scopeRange at a position?

From @billti on November 1, 2015 6:10

The API call document.getWordRangeAtPosition(position) appears to use its own definition of a word. For example, my tmLanguage defines attrib-name as a token/scope, yet getWordRangeAtPosition appears to break this into 2 words on the - character.

How can I get token ranges at a position based on my custom syntax? (And it would be really useful if I could get the scope name that goes along with it too).

Copied from original issue: Microsoft/vscode-extensionbuilders#76

[savetheclocktower]

There’s still no extension API that returns text-mate scopes. Several reasons for that one of them that we don’t want that extensions start depending on a particular tokenizer grammar.

I feel silly responding to a four-year-old comment, but it is the last “official” word on this issue, so here I go.

VSCode is the third major code editor to borrow TextMate’s grammar system, and I wonder if they all thought that its scope names were simply an implementation detail of its grammars. Quite the opposite — a lot of thought went into this system, since it was also used as the basis for much of TextMate’s customizability.

Scope names aren’t just hooks for syntax highlighting. TextMate commands are tightly woven to the semantics of scope names. You can have the same key combination perform different commands based on scope, so that your command doesn’t monopolize a hotkey for something that (say) only works when the cursor is within a string. Conversely, you can define a command that behaves identically across different languages because it hooks into the presence of a generic scope name. This is how TextMate recognizes URLs across multiple contexts — inside HTML files, inside Markdown files, inside code comments regardless of language — and implements a single “open this URL in my browser” command that works identically across all of them.

Even after moving from TextMate to Atom several years ago, I was able to keep almost all of my ornery customizations because Atom allowed me to inspect scopes at the cursor. I define a command in my Atom init-file whose only purpose is to interpret <kbd>Enter</kbd> on my num pad and delegate it to one of three other commands based on what scope I’m in. If I migrate to VSCode in the future, it’ll be a reluctant migration if this issue is still open.

TextMate’s scope naming conventions are middleware. VSCode could move to tree-sitter grammars tomorrow without breaking anyone’s syntax coloring themes; it’d just need to map tree-sitter token names to existing scope names. If a “get all scopes at position X” API existed in VSCode, and I relied upon it when writing an extension, that extension would keep working in a future version of VSCode that no longer supported TM-style grammars but kept TM’s scope naming conventions.

You may think, “Yes, but we don’t want to make these naming conventions permanent! That’s the whole point!” To which I’d ask: what would you replace them with, and why? Is there something that the existing naming conventions can’t do? Is there a compelling reason to invent something new that would justify the amount of community effort it would take to adopt a new scheme? Would it make a migration toward tree-sitter grammars easier or harder if syntax coloring themes had to support two different naming systems at once?

As the comments on this issue illustrate, not having this functionality doesn’t remove the need for extensions to know this information; it just means those extensions have to use imperfect workarounds. And it results in tighter coupling to TextMate grammars than would otherwise be necessary, since those workarounds need to know the grammar’s implementation details to reproduce the result.

I hope you’ll consider this feature request sometime soon; it’d be a huge customizability win.

[ImUrX]

this issue would be so useful and is one of the oldest thats still open