Stale semantic tokens across several files

Meta: this is a complex issue touching VS Code APIs, LSP, the way LSP is bound to VS Code API in vscode-languageserver-node, and the way rust-analyzer implements LSP. It feels like there’s some bug/design issue in one of the components, but its hard to pinpoint the culprit exactly.

The problem is, if rust-analyzer just directly implements the spirit and letter of the LSP protocol, the user can observe inconsistent syntax highlighting:

https://user-images.githubusercontent.com/1711539/127041106-f6e2c10b-b5f4-4795-af50-8359238aa0ab.mp4

In rust-analyzer, syntax highlighting of a file depends not only on the contents of the file itself, but also on the global state of the project, including contests of other files. A direct implementation of LSP results in an artifact, where modifications to file A don’t cause highlights for file B to be recomputed.

The way we work-around this is by issuing a refresh server->client notification on every change:

https://github.com/microsoft/language-server-protocol/issues/1163#issuecomment-886936431

This, strictly speaking, violates the protocol (refresh should only be send for “project configuration” changes, like a user adding a new dependency to Cargo.toml). It also just feels silly: a client sends us a DidChangeTextDocument event, and the server immediately and unconditionally responds with /refresh, asking the client to send a bunch of follow-up requests.

Not that this isn’t a big problem in practice – most of the time the results are OK in practice. But it does seem that the overall model of handling state has some gap.

Note also that semantic tokens isn’t the only API that’s affected – the same effect can be observed for code lens, for example.

[jrieken]

Wrt open document and visible editor and the lack of “non-visible editors”: we are actively working on exposing the tab model to extensions, something like vscode.window.tabs. That should be helpful here as well

[alexdima]

is there any reason why we don’t pull for semantic tokens for all files that are visible in the editor?

To the editor layer itself, the relationship between these two text buffers is not clear. To us, looking at the screen recording above, it is obvious that one file imports a symbol from the other, but to the editor widget, the left-hand-side could have been a Rust file, and the right-hand-side could have been a Markdown file, the two editor instances don’t really know about each-other.

We currently don’t have a way to track that there is a relationship between these files, that editing a file A should result in updating editor elements for file B. We only have the languageId that could be used to know that there is a connection here.

@jrieken I think this touches on many language features (e.g. breadcrumbs, codelenses, folding ranges), as well as semantic tokens. I would not want to implement something only for semantic tokens and deviate from the other providers. I think all providers should behave somewhat equally here. Should we implement that if two or more text buffers are displayed, and one is edited, then all others of the same language id will refresh their UI. To clarify, if there is a codelens provider for TS, it should only refresh codelenses in visible TS files when typing in other TS files (maybe with a longer debounce time if the file is not focused).