Razor LSP server appears to be able to run requests on wrong document version

Describe the bug:

While investigating Semantic tokens can occasionally ask for wrong range of document after ctrl+Z, filed against VS LSP client, I observed what appears to be a potential race condition in the Razor LSP server that may be responsible for 1508587.

@NTaylorMullen, @ryanbrandenburg could this be the root cause of the infamous Disco Colors issue? I fixed all known VS LSP client repros of this bug in 17.1, but this customer is reporting that it still repros. The behavior I’m observing suggests that LSP requests in Razor could occasionally run against the wrong document version.

Version used:

e.g. VS2022 17.3 Preview 1

To reproduce:

1. Create a new Blazor server app
2. Open FetchData.razor
3. Replace the content with:

@page "/fetchdata"

<PageTitle>Weather forecast</PageTitle>

@using BlazorApp70.Data
@inject WeatherForecastService ForecastService

<strong>
    @DateTime.Now.ToString()
</strong>


<div>
    <strong>
        @DateTime.Now.ToString()
    </strong>
</div>


@{
    var abc = true;

    void Bar()
    {
        
    }
}


<content>
    <div>
    </div>
</content>
<strong>
    @DateTime.Now.ToString()
</strong>

<h1>Weather forecast</h1>

<p>This component demonstrates fetching data from a service.</p>

@if (forecasts == null)
{
    <p><em>Loading...</em></p>
}
else
{
    <table class="table">
        <thead>
            <tr>
                <th>Date</th>
                <th>Temp. (C)</th>
                <th>Temp. (F)</th>
                <th>Summary</th>
            </tr>
        </thead>
        <tbody>
            @foreach (var forecast2 in forecasts)
            {
                <tr>
                    <td>@forecast2.Date.ToShortDateString()</td>
                    <td>@forecast2.TemperatureC</td>
                    <td>@forecast2.TemperatureF</td>
                    <td>@forecast2.Summary</td>
                </tr>
            }
        </tbody>
    </table>
}


@code {
    private WeatherForecast[]? forecasts;

    protected override async Task OnInitializedAsync()
    {
        forecasts = await ForecastService.GetForecastAsync(DateTime.Now);
    }
}

It should be 80 lines long.

1. On line 30 (<content>put your cursor after the<c` and hit enter.
2. Hit ctrl + z (undo).
3. Repeat several times. If done quickly enough, an ArgumentOutOfRangeException is thrown from the Razor semantic tokens handler.

Explanation

This was originally filed as a VS LSP client bug. Inspecting both components, I don’t think this sort of bug should be possible in the LSP client.

• We have one thread that dispatches LSP messages per client per document.
• This thread is responsible for updating the text version via didChange
• Semantic Tokens goes through that same thread
• This thread keeps a record of the most recently sent ITextSnapshot.
• Any requests sent have their Positions and Ranges translated to that snapshot.

This process automatically realigns spans with the bounds of the translated snapshot, and there’s only one thread, so there shouldn’t be many opportunities to race.

Background

• LSP is a ‘well-ordered’ protocol - each message is assumed to apply to the state of the world after the in-order application of previous messages.
• An implication of this is that a correctly behaved server must guarantee that any textDocument/didChange notifications are fully applied before executing any subsequent requests.
• In other words, in didChange A => B, didChange B => C, didChange C => D, semanticTokens, the semantic tokens request should run on version D.
• Inspecting Razor didChange handler, however, it appears that it could potentially run on version C, as seems to be happening in this scenario.

Existing Razor Support

Existing Razor support attempts to enforce the above requirements:

• There is a single ‘dispatcher’ thread which is used to execute didChange notifications and document version info lookups.
• The dispatcher thread is the right idea and is mostly effective, however, it is executing handlers as a series of disjoint continuations. It seems like these could potentially interleave, violating the second bullet from the ‘Background’ section.

Potential Root Cause

Razor code is yielding in the middle of application of a didChange notification.

It looks like this introduces the possibility that a semantic tokens request that arrives after a didChange could become interleaved with that didChange, causing it to get handled before the text version it is supposed to run on is recorded in Razor’s project service.

Supporting Evidence

I added static variables to the didChange message handler in the first dispatcher continuation (captures the most recently received text version) and in the last dispatcher continuation (captures the most recently ‘committed’ text version that is visible for use by the semantic tokens handler.

I then added code in the Semantic Tokens handler that launches the debugger any time received and committed version numbers are different in the middle of the part of the semantic tokens handler that runs on the dispatcher thread, and it was hit immediately.

While this isn’t definitive proof of a bug (perhaps the received version is from a didChange immediately following the erroring semantic tokens call), it is suspicious, as it demonstrates that there are cases where a semantic tokens request’s version can be established on the dispatcher thread interleaved between continuations of the didChange handler.

[gundermanc]

FYI: Thanks to a repro from the great folks on the debugger team, I did end up finding a race condition on the LSP client side where in extremely obscure cases we can send request messages out of sequence.

I think the aforementioned Razor code is still suspect, so the work done to improve it is a worthwhile investment, but there are fixes coming to VS in 17.5 that should improve resilience of various features: https://devdiv.visualstudio.com/DevDiv/_git/VSLanguageServerClient/pullrequest/426457

[ryanbrandenburg]

This is hypothetically resolved now that CLaSP is merged. I will close and we can re-open if the issue re-cures.